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5th  Annual  CMMI  Technology  Conference  and  User  Group 
Denver,  CO 
14-17  November  2005 


Agenda 


Monday.  14  November  2005 


Tutorial  Tracks 
Track  1 : 

•  Calculating  CMMI-Based  Return  on  Investment  (ROI):  Why,  When,  What,  How?,  Mr.  Rolf  W.  Reitzig,  Cognece,  Inc. 

•  A  Practical  Guide  to  Implementing  Levels  4  and  5,  Mr.  Rick  Hefner,  Northrop  Grumman  Corporation 

Track  2: 

•  Agile/Lean  Workshop,  Mr.  Jeffrey  Dutton,  Jacobs  Sverdrup 

•  Mr.  Tim  Kasse,  Kasse  Initiatives,  LLC: 

1 .  Leveraging  ITIL  Services  (Suppport  and  Delivery)  Capability  and  Maturity  with  the  CMMI 

2.  Service  Management  “A  Process  Led  Approach” 

3.  ITIL  IT  Infrastructure  Library  Overview 

4.  Overview  of  Service  Support  &  Service  Delivery  Functions 

5.  Configuration  Management  &  Change  Management 

6.  “Service  Support”  Change  Management 

7.  “Service  Support”  Configuration  Management 

8.  Configuration  Management  &  Change  Management  ITIL  -  CMMI 

9.  ITIL  Process  Maturity  Self-Assessment  &  Action  Plan 

Track  3: 

•  The  Look  and  Feel  of  a  Successful  CMMI  Implementaiton,  Mr.  Tim  Kasse,  Kasse  Initiatives,  LLC 

•  How  to  Define  CMMI  Based  Process  That  are  Short  and  Usable  and  Using  a  Process  Measurement  Framework  to  Successfully  Achieve  Measurable  Results, 
Mr.  Timothy  G.  Olson,  Quality  Improvement  Consultatnts,  Inc. 

Track  4: 

•  Using  Simulation  to  Support  Better  Management  Decisions,  Dr.  David  M.  Raffo,  Portland  State  University 

•  Institutionalizing  Resource  Planning  and  Management  Part  I  and  Part  II,  Mr.  Donald  A.  Borcherding,  NexSummit,  LLC 


Track  5: 

•  The  CMMI  V1.2  ...  A  Tutorial,  Mr.  David  M.  Phillips,  SEI 


Track  6: 

•  Integrated  Porject  Management  (IPM)  -  The  CMMI  and  Collaborative  Product  Develop  and  Requirement  Engineering:  A  Practicial  Approach  to  Modeling 
and  Managing  Requirements,  Mr.  William  J.  Deibler,  II  Software  Systems  Quality  Consulting  -  SSQC 

Tuesday.  15  November  2005 

General  Sessions 

LTG  Joseph  Yakovac,  USA,  Miliatry  Deputy  Office  of  the  Secretary  of  the  Army,  Acquisition,  Logistics  &  Technology 
Executive  Panel:  "How  Has  CMMI  Improved  Our  Program  &  Project  Performance  —  Or  Has  It?": 
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Mr.  Mark  Schaeffer,  Director,  Systems  Engineering,  OUSD(AT&L)  Defense  System  and  OSD  Sponsor,  CMMI 

•  Mr.  Dev  Banerjee,  Division  Director,  Systems  &  Flight  Engineering,  Boeing  Integrated  Defense  Systems 

•  Mr.  John  Evers,  Raytheon  Processes  Program  Manager,  Raytheon  Common  Engineering  Process  Program 

•  Brig  Gen  Gary  Salisbury,  USAF  (Ret),  Executive  Director  Business  Development,  Defense  Mission  Systems,  Northrop  Grumman  Mission  Systems 
Lunch:  CMMI  State  of  the  Model,  Mr.  Bob  Rassa,  Raytheon;  Mr.  Clyde  Chittister,  SEI 

Technical  Sessions 

Track  1 :  CMMI  Process  Improvement 

•  CMMI/ISO  -  "Can't  we  all  just  get  along?",  Mr.  Dale  R.  Spaulding,  The  Boeing  Company 

•  Real  World  Application  of  IEEE  Software  Engineering  Standards  to  CMM/CMMI  Software  Process  Improvement  Initiatives,  Ms.  Susan  K.  Land,  Norhtrop 
Gurmman  IT/TASC 

•  The  CMMI  Product  Suite  and  International  Standards  —  An  Update,  Mr.  David  H.  Kitson,  SEI 
Track  2:  Practical  Guidance 

•  Verification  in  CMMI  using  Peer  Reviews  -  Presentation  and  Paper,  Ms.  Jeanne  H.  Balsam,  Georgia  Tech  Research  Institute 

•  Process  QA  in  the  Information  Age:  Keep  it  Light!,  Hillel  Glazer,  Entinex,  Inc 

•  Defect  Datat  and  Configuation  Management,  Ms.  Julie  E.  Schmarje,  Raytheon  Company 

Track  3:  Appraisals 

•  Wasted  Days  and  Wasted  Nights  -  Leveraging  Your  Appraisal  Team  as  a  Resource,  Dr.  Timothy  J.  Davis,  Raytheon  Missile  Systems 

•  Building  a  Credible  SCAMPI  Appraisal  Representative  Sample,  Mr.  Robert  L.  Moore,  III.,  Business  Transformation  Institute,  Inc. 

•  Top  10  Signs  You're  Ready  (or  Not)  for  an  Appraisal,  Mr.  Gary  Natwick,  Harris  Corporation 

Track  4:  ROI  &  Benefits  of  CMMI 

•  Measuring  Performance:  Evidence  About  the  Results  of  CMMI,  Ms.  Diane  Gibson,  SEI 

•  Prioritizing  Process  Improvement  Strategies  in  CMMI  to  Optimize  Business  Objectives,  Dr.  Aldo  Dagnino,  ABB,  Inc.  US  Corporate  Research 

•  Implementing  a  Plan  for  Controlling  ROI  for  CMMI  Process  Improvement,  Mr.  J.  M.  Perry,  BAE  Systems 

•  Lessons  Learned  in  the  Engineering  of  Process  Performance  Models  on  the  Journey  to  Higher  Maturity  Levels,  Mr.  Dr.  Mary  Anne  Herndon,  SAIC 
Track  5:  Acquisition  /  High  Maturity 

•  Getting  Lost  on  the  Way  to  Level  5,  Ms.  Kathy  King,  The  Center  for  Systems  Management 

•  Understanding  Why?,  Mr.  David  N.  Card,  Q-Labs 

Track  6:  Transitioning  to  CMMI 

•  Migrating  Best  Practices  Within  an  Organization:  Experiences  in  Adapting  CMMI  Policies  and  Procedures  Used  in  One  Part  of  a  Business  to  Another,  Mr. 
Scott  Sherrill,  Georgia  Tech  Research  Institute 

•  An  Enterprise  Wide  CMMI  Implementation  at  Accenture,  Ms.  Sarah  S.  Bengzon,  Accenture 

•  Stakeholder  Identification  and  Involvement  in  the  CMMI,  Mr.  James  R.  Armstrong,  Systems  and  Software  Consortium 

•  Ensuring  the  Right  Process  is  Deployed  Right:  Synchronizing  Process  Checkpoints  with  Business  Rhythm,  Ms.  Joan  Weszka,  Lockheed  Martin  Corporation 
Track  7:  CMMI  for  Small  Projects  anhd  Organizations 

•  Making  PPQA  Work  on  Small  Projects  Presentation  and  Paper,  Ms.  Jean  M.  Swank,  Georgia  Tech  Research  Institute 

•  Does  Size  Matter  in  CMMI  Implementation  or  Was  Yoda  Wrong?,  Mr.  Paul  H.  Meyers,  SAIC 


Wednesday.  16  November  2005 


Technical  Sessions 


Track  1:  CMMI  Process  Improvement 

•  A  Change  Agent  in  a  Level  1  Organization;  How  to  Survive  in  a  Hostile  Environment,  Mr.  Andrew  Cordes,  ABB  -  United  States  Corporate  Research  Center 

•  "Sound  Systems  Engineering  Using  CMMI,  Mr.  Michael  T.  Kutch,  Jr.,  SPAWAR  -  Charleston 

•  Using  CMMI  to  "Dig  Out"  from  an  Ad  Hoc  Development?,  Mr.  Donald  A.  Borcherding,  NexSummit,  LLC 

•  Strategic  Planning:  Selling  a  CMMI-Based  Improvement  Effort  to  Senior  Management,  Dr.  Aldo  Dagnino,  ABB,  Inc.,  US  Corporate  Research 

•  Enterprise  Process  Intergration  within  the  Space  and  Airborne  Systems  Business  Area  of  Raytheon,  Mrs.  Deana  A.  Seigler,  Raytheon  Company 

•  Interpreting  the  CMMI:  It  Depends!,  Mr.  Rick  Hefner,  Northrop  Grumman  Corporation 

•  CMMI  as  Safeguard  Against  Software  Entropy:  A  Manager's  Perspective,  Dr.  Thomas  F.  Christian,  Jr.,  402  SMXG 

•  "Its  how  big?  How  will  you  deploy  it  without  killing  my  team  and  my  program?",  Mr.  William  Borkowski,  Jr.,  Raytheon  Missile  Systems 


Track  2:  Practical  Guidance 

•  Are  You  Making  the  Most  of  Your  Project  Schedules?,  Ms.  Susan  Byrnes,  PMP,  Natural  SPI,  Inc. 


5th  Annual  CMMI  Technology  Conference  and  User  Group.html[8/25/2016  1:39:09  PM] 


Untitled  Document 


•  Keeping  the  Team  Motivated  for  Success  and  “Barrier  Busting”  -  Obtaining  Active  Leadership  Support,  Mr.  Michael  D.  Scott,  Raytheon  Missile  Systems 

•  Using  a  Level  3  Process  to  Achieve  CMMI  Level  3,  Mr.  Stephen  Ross,  Raytheon  Company 

•  Accelerating  Process  Improvement  through  Collaboratio:  The  NAVAIR  Systems  Process  Improvement  Community  of  Practice,  Ms.  Katie  Smith,  Naval  Air 
Systems  Command 

•  What  the  CMMI  Doesn't  Say  About  Training  (But  Should!),  Sree  Yellayi,  Siemens  Corporate  Research 

•  CMMI  CP  2.8  Interpretation  and  Implementation:  Is  This  Practice  Just  About  Numbers?,  Mr.  Lester  Stamnas,  Norausky  Process  Solutions,  Inc. 

•  Creating  Helpful  process  Directives,  Mr.  Kenneth  I.  Weinberg,  Raytheon  Company 


Track  3:  Appraisals 

•  Lessions  Learned  in  Helping  Large  and  Small  Organizations  Prepare  for  their  First  Appraisal,  Mr.  Robert  J.  Pomietto,  Center  for  Systems  Management 

•  Behind  Closed  Doors,  Mr.  Tom  G.  Lienhard,  Raytheon  Missile  Systems 

•  CMMI  Appraisal  Results:  The  Shocking  Truth  Revealed  and  Lead  Appraisers  Gone  Wild,  Ms.  Margaret  A.  Glover,  SEI 

•  Improving  Document  Reviews  for  Appraisals,  Mr.  Kent  McClurg,  Raytheon  Company 

•  Finding  CMMI  Compliant  Artifacts  and  a  Needle  in  a  Haystack,  Adrio  J.  DeCicco,  Raytheon  Company 

•  Lessons  Learned  and  Best  Practices  for  Evidence  Collection  in  Preparation  for  a  SCAMPI  Appraisal,  Mr.  Ben  Berauer,  Raytheon  Company 

•  Maximizing  Value  for  SCAMPI(SM)  Preparation,  Ms.  Joan  Weszka,  Lockheed  Martin  Corporation 


Track  4:  ROI  &  Benefits  of  CMMI 

•  Evaluating  the  Impact  New  Tools  and  Technologies  Using  Simulation,  Dr.  David  M.  Raffo,  Portland  State  University 

•  The  ROI  Dashbord  (c):  Understanding  the  Benefits  of  CMMI,  Mr.  Thomas  L.  McGibbon,  ITT  Industries,  AES 

•  Quality  Assurance  Involvement  Compared  to  Program  Results,  Ms.  Jill  Brooks,  Raytheon  Company 

•  Rapidly  Achieving  Measurable  ROI  Using  Early  Defect  Detection,  Mr.  Timothy  G.  Olson,  Quality  Improvement  Consultants,  Inc 

•  CMMI  Process  Improvement:  Its  not  a  technical  problem,  its  a  people  problem,  Mr.  Rolf  W.  Reitzig,  Cognence,  Inc. 

•  A  Project's  Perspective  of  a  CMMI  Level  5,  Mr.  Warren  Scheinin,  Northrop  Grumman  Corporation 

•  Achieving  the  Promised  Benefits  of  CMMI,  Dr.  Rick  Hefner,  Norhtrop  Grumman  Corporation 

•  Measuring  Economic  Benefits  of  Process  Improvement  in  CMMI  Level  1  Organization,  Dr.  Aldo  Dagnino,  ABB,  Inc.,  US  Corporate  Research 


Track  5:  High  Maturity 

•  Logarithms  Can  Be  Your  Friends:  Controlling  Peer  Review  Cost?,  Dr.  Richard  L.  W.  Welch,  Northrop  Grumman  Corporation 

•  Journeys  on  the  Road  to  Level  5,  Mr.  Joseph  V.  Vanderville,  Northrop  Grumman  Corporation 

•  Lessons  Learned  on  the  SCAMPI  Road  to  CMMI-Software  Level  5,  Mr.  Joseph  N.  Frisina,  BAE  Systems 

•  Merging  Measurement  in  Mature  Companies  -  A  Success  Story  of  Measurement  Process  Integration,  Ms.  Sharon  Rohde,  Lockheed  Martin  IS&S 

•  The  Road  to  Process  Improvement  Successes:  CMMI  Level  5/ISO  9001-2000  Business  Model,  Mrs.  Debra  S.  Roy,  BAE  Systems,  National  Security  Solutions 

•  Reducing  Variation  at  Each  CMMI  Maturity  Level?,  Mr.  Timothy  Kasse,  Kasse  Initiatives,  LLC 

•  Ways  to  Ensure  the  Culture  Supports  Level  5,  Mr.  Warren  Scheinin,  Northrop  Grumman  Corporation 

•  Analyzing  Defects  Can  Tell  a  Story  About  a  Company?,  Ms.  Diane  A.  Mitzukami-Williams,  Northrop  Grumman  Corporation  Mission  Systems 


Track  6:  Transition  to  CMMI 

•  Combining  Six  IPTS  and  Transitioning  to  CMMI,  Ms.  Judy  Overhouser-Duett,  NAVAIR 

•  How  to  Transition  Models  and  Disciplines  -  Looking  for  Transition  in  all  the  Wrong  Places,  Ms.  Lori  G.  Smailes,  TYBRIN  Corporation 

•  Using  SW-SMM  SQA  Independent  Verification  as  a  First  Step  for  the  Transition  to  CMMI,  Mr.  Alfredo  N.  Tsukumo,  CenPRA-Centro  de  Pesquisas  Renato 
Archer 

•  Service  Extensions  to  the  CMMI,  Mr.  Craig  R.  Hollenbach,  Northrop  Grumman  Corporation 

•  Applying  CMMI  to  Services,  Mr.  Juan  C.  Ceva,  Raytheon  ITSS 

•  Management  Challenges  &  Lessons  Learned  Implementing  CMMI  in  a  Services  Environment,  Mr.  Thomas  E.  Zience,  BAE  Systems  Information  Technology 

•  CMMI  vl.l  for  a  Service  Oriented  Organization,  Mr.  Steven  K.  Hall,  Raytheon  Corporation 


Track  7:  Measurement 

•  Software  Size  Growth  and  Uncertainity  -  Both  Affect  Estimate  Quality  and  Project  Risk  Presentation  and  Paper,  Mr.  Michael  A.  Ross,  Galorath,  Inc. 

•  Building  an  Automated  System  to  Support  Measurement  in  CMMI,  Dr.  Richard  Hayden,  Pragma  Systems  Corporation 

•  Team  of  Three  -  How  to  Get  Program,  Functional  and  Process  Management  Working  Together,  Mr.  Mark  A.  Marsh,  Raytheon  Company 

•  Parametric  Project  Monitoring  and  Control:  Performance-Based  Progress  Assessment  and  Prediction  Presentation  and  Paper,  Mr.  Michael  A.  Ross,  Galorath, 
Inc. 

•  Measuring  and  Estimating  Process  Performance,  Dr.  Richard  D.  Stutzke,  SAIC 

Thursday.  17  November  2005 


Technical  Sessions 


Track  1 :  CMMI  Process  Improvement 

•  “Barrier  Busting”  -  Obtaining  Active  Leadership  Support,  Mr.  Michael  D.  Scott,  Raytheon 

•  Don’t  Write  the  Wrong  Processes!,  Ms.  Suzanne  B.  Zampella,  The  Center  for  Systems  Management 
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•  Contrasting  CMMI  Contrasting  CMMI  and  the  PMBOK,  Mr.  Wayne  Sherer,  Anteon  Corporation 

•  Being  Customer  Oriented,  Mr.  Tim  Kasse,  Kasse  Initiatives,  LLC 

•  Learning  from  Lessons  Observed  -  Mitigating  Resistance  to  Process  Improvement,  Mr.  Bob  Norris,  National  Geospatial-Intelligence  Agency 


Track  2:  Practical  Guidance 

•  Supplier  Management  Strategy  Considerations  with  CMMI,  Mr.  Rick  Hefner,  Northrop  Grumman  Corporation 

•  Simplifying  Process  Tailoring  To  Enhance  Project  Execution,  Mr.  Howard  T.  Kaplan,  Raytheon  Company 

•  CMMI  and  agile:  a  High  Tech  R&D  Success  Story,  Mr.  Gene  Miluk,  SEI 

•  How  to  Incorporate  “Lessons  Learned”  for  Sustained  Process  Improvements,  Mr.  Anil  K.  Midha,  BAE  Systems 

•  Data  Management:  The  Hidden  Enabler  or  (The  Key  Data  and  Work  Product  Integrator),  Mr.  Lester  Stamnas,  Norausky  Process  Solutions 


Track  3:  Appraisals 

•  Techniques  for  Shortening  the  Time  and  Cost  of  CMMI  Appraisals,  Mr.  Sam  Fogle,  Systems  and  Software  Consortium,  Inc. 

•  Using  Classified  Programs  in  CMMI  Appraisals,  Mr.  Kenneth  I.  Weinberg,  Raytheon  Company 

•  The  Best  Intentions  of  SCAMPI  VI .  1 :  What  We  Meant  and  What  Some  People  Heard,  Mr.  Will  Hayes,  SEI 

•  A  Quantitative  Comparison  of  SCAMPI  A,  B,  and  C,  Mr.  Dan  Luttrell,  Northrop  Grumman  Mission  Systems 

•  Performing  Consistent  Appraisals  in  a  Global  Organization,  Ms.  Jeanine  Courtney-Clark,  Integrated  System  Diagnostics,  Inc 


Track  4:  ROI  &  Benefits  of  CMM  I  /  SCAMPI  B/C 

•  The  Effects  of  CMMI  on  Program  Performance,  Mr.  Joseph  P.  Elm,  SEI 

•  Squeezing  Variation  for  Profit,  Mr.  Donald  R.  Corpron,  Northrop  Grumman  Corporation 

•  Process  In  Execution  Review  (PIER)  and  the  SCAMPI  B  Method,  Ms.  Lorraine  J.  Adams,  SEI 

•  Planning  a  SCAMPI  C  Appraisal  from  a  Strategic  Perspective,  Mr.  John  P.  Kennedy,  The  Mitre  Corporation 

•  Critical  Path  SCAMPISH  Getting  Real  Business  Results  from  Appraisals,  Mr.  Michael  J.  West,  Natural  SPI,  Inc. 

•  Using  SCAMPI  C  for  Collective  Improvement  Across  a  Multi-Business  Program,  Mr.  Oktawian  Nowak,  Motorola,  Inc. 


Track  5:  High  Maturity 

•  A  Statistical  Approach  to  Product  Quality  Assurance,  mr.  Randall  J.  Varga,  BAE  Systems 

•  The  Key  to  a  High  Maturity  Rating  is  -  ORGANIZATION,  Mrs.  Karen  M.  Pelletier,  Northrop  Grumman  Corporation 

•  Paladin  Drives  Forward  To  CMMI®  Maturity  Level  5,  Mr.  Victor  Elias,  M.S.,  Armament  Software  Engineering  Center,  US  Army 

•  Business  Improvements  Achieving  CMMI®Level  5  at  SAIC:  Who  Keeps  Moving  My  Process?,  Ms.  Sharon  Cobb  Flanagan,  SAIC 

•  Extending  CMMI  Level  4/5  Organization  Metrics  Beyond  Software  Development,  Ms.  Linda  R.  Brooks,  Northrop  Grumman  Corporation 


Track  6:  CMMI  Extensions 

•  Capability  Maturity  Model  Integration  (CMMI®)  Tailoring  for  an  IT/MS  Services  Environment,  Ms.  Stacy  Savage,  BAE  Systems  Information  Technology 

•  Adapting  CMMI  for  Acquisition  Organizations:  A  Preliminary  Report,  Dr.  Hubert  Hofmann,  General  Motors 

•  How  to  Become  Your  Customer’s  Software  Provider  of  Choice,  Mr.  David  Herron,  DCG,  Inc. 

•  Space  and  Missile  Systems  CenteSpace  and  Missile  Systems  Center,  Mr.  Keith  Wright,  SPARTA,  Inc. 

•  Software  Outsourcing  with  CMMI,  Dr.  John  W.  Mishler,  SEI 


Track  7:  Systems  Engineering 

•  Sound  Systems  Engineering  using  CMMI®,  Ms.  Sandee  D.  Guidry,  TECHSOFT 

•  Systems  Engineering  Influence  Throughout  the  CMMI,  Mr.  Tim  Kasse,  Kasse  Initiatives,  LLC 

•  Future  of  System  and  Software  Engineering  Project  Management  and  the  CMMI,  Dr.  Kenneth  E.  Nidiffer,  Systems  and  Software  Consortium 
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November  14  -  17,  2005 
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CMMI  is  registered  in  the  US  Patent  &  Trademark  Office  by  Carnegie  Mellon  University 


Sunday,  November  13,  2005 


12:00  PM  -4:00  PM 

Registration  for  Conference  and  Tutorial 

Atrium 

Monday,  November  14,  2005 


7:00  AM  -5:00  PM 

Tutorial  Registration  ($200  Tutorial  Fee) 

Atrium 

7:00  AM  -8:00  AM 

Continental  Breakfast  (Tutorial  Attendees  Only) 

Atrium 

8:00  AM  -5:00  PM 

CMMI  Tutorial  Tracks  (Tutorial  Attendees  Only) 

See  Following  Pages 

12:00  PM  - 1:00  PM 

Lunch  (Tutorial  Attendees  Only) 

Grand  Mesa  ABC 

5:00  PM  -  6:30  PM 

Reception  (All  CMMI  Conference  Attendees) 

Display  Area 

Tuesday,  November  15,  2005 


7:30  AM  -8:30  AM 

Registration  and  Continental  Breakfast 

Atrium 

8:30  AM  -8:45  AM 

Opening  Remarks 

Grand  Mesa  DEF 

8:45  AM  -9:30  AM 

Grand  Mesa  DEF 

Session  A 

LTG  Joseph  Yakovac,  USA,  Military  Deputy,  Office  of  the  Secretary  of  the  Army, 
Acquisition,  Logistics  &  Technology 


9:30  AM  - 10:00  AM 

Break 

Atrium 

10:00  AM  -  12:00  PM 

Session  B 

Executive  Panel  -  “How  Has  CMMI  Improved  Our  Program  &  Project 
Performance  -  Or  Has  it?” 

Moderator: 

Mr.  Mark  Schaeffer,  Director,  Systems  Engineering,  OUSD(AT&L) 

Defense  Systems  and  OSD  Sponsor,  CMMI 

Grand  Mesa  DEF 

Panelists: 

Mr.  Dev  Banerjee,  Division  Director,  Systems  &  Flight  Engineering,  Boeing 
Integrated  Defense  Systems 

Mr.  John  Evers,  Raytheon  Processes  Program  Manager,  Raytheon  Common 

Engineering  Process  Program 

Brig  Gen  Gary  Salisbury,  USAF  (Ret),  Executive  Director,  Business  Development, 
Defense  Mission  Systems,  Northrop  Grumman  Mission  Systems 


CONFERENCE  AGENDA 


CONFERENCE  AGENDA 


12:00  PM  -  1:30  PM 

Lunch 

CMMI  -  State  of  the  Model 

Mr.  Bob  Rassa,  Raytheon;  Mr.  Clyde  Chittister,  SEI 

Grand  Mesa  ABC 

1:30  PM  -5:00  PM 

Technical  Sessions 

See  Following  Pages 

3:00  PM  -  3:30  PM 

Break 

Display  Area 

5:00  PM  -  6:30  PM 

Reception 

Display  Area 

Wednesday,  November  16,  2005 


7:00  AM -8:00  AM 

Registration  and  Continental  Breakfast 

Atrium 

8:00  AM -5:00  PM 

Technical  Sessions 

See  Following  Pages 

9:30  AM  -  10:30  AM 

Break 

Display  Area 

12:00  PM  -  1:30  PM 

Lunch 

Conference  Best  Paper  Awards 

Grand  Mesa  ABC 

3:00  PM  -  3:30  PM 

Break 

Display  Area 

Thursday,  November  17,  2005 


7:00  AM -8:00  AM 

Registration  and  Continental  Breakfast 

Atrium 

8:00  AM  -2:30  PM 

Technical  Sessions 

See  Following  Pages 

9:30  AM  -  10:30  AM 

Break 

Display  Area 

12:00  PM  - 1:00  PM 

Lunch 

Grand  Mesa  ABC 

2:30  PM 

Conference  Adjourns 

RECEPTION  IN  DISPLAY  AREA  (5:00  PM)  (ALL  ATTENDEES) 


<0 

"O 

C 

o 


o 


gS  o 

5?  d 

If ! 

S 

c 

1 1  S> 


S|8I 

Q  o  "35  ~  is 

T'jQSS 


BREAK  (2:45  PM)  (TUTORIAL  ATTENDEES  ONLY) 


lia 

gig 

I  Is 

\  co  to  v: 


05 


O  05 

€4 

2^ 

°*  S 

fg  <S 


o 

£8 

|W 


S  “  5:  3 

®2>  = 
■Si§  jio 

050  -Q  o 

>5|  Q.$* 

42^42^ 

C  fi  C  ^  3 
O  O  E  Qf 

S.O  * 
g  to  £  =3  E 
‘3  8'5ia 

CO  O'Q.O"  .  (0 

O  O  Q.  Q) 
t-  q;  >q;  Q;  §  to 


LUNCH  (12:00  PM)  (TUTORIAL  ATTENDEES  ONLY) 


o 

55? 


‘Sg’ 

CO  -S 


II 

li 


O  o 

c-S 

i?-2  -  , 

l§pa 

•I?  1st 


oS  5 

bS-IIM 

SI  I  Q^M 


BREAK  (9:45  AM)  (TUTORIAL  ATTENDEES  ONLY) 


05 

.J  « 


?-S8 

Sag 

■g§* 
o  E  E 
J®i: 

0  05  Q  . 


Track  1 

Grand 
Mesa  D/E 


Track  2 

Grand 
Mesa  F 


Track  3 

Highlands 


5 

>iC  ,05 

o-Q 

^  2 
|  5 
22 

|*5 

§42 

u.§ 

c  2 

i| 

So 

I! 

OO 

3«c* 
CO  ^ 
05  "E 

§5 

CD  ^ 

tl 

O  nt 

OJ  (0 

42  P  S 

O)®' 

■s  [S  O 

«5-S  C5 

3  K 

CO  .  o 

«q;  co  3  05  c 

Track  5 

Mesa  Verde 


Track  6 

Wind  River 


> 

<0 

"D 

(/) 

O 

3 


o 


RECEPTION  IN  DISPLAY  AREA  (5:00  PM  -  6:30  PM) 


CD  1  LU 

wfec 

T?"0  O 
^  to  co 
-oiSrt 
OC/D^ 

Q-roa: 

2.|| 

O  CQ 

77  CD  CD 


So 

-S2  S 

LU^  O 
LLJ  CD  O 

m-OQ. 

^  CO  CD 
^  C/5  CO 

Jd>| 
O  few 
o-l® 

<i| 

■DLU^ 
«  (  ) 
fe  05< 

^  5© 

^«J| 

Q  CD 

cnjQ^WO 


■c 

C0_§ 


.55 

.ti  ccW 

a-ig 

c*H 
CD  c  — 

III 


(0 

(0  c 

CD  O 

s“ 

Q-  § 

?  I 

<0  § 

§  8 
it 


> 

"O 

CO 

CD 


C/5 


n:  ro 

CD  _ 

$31 


C/5  ■  — 

c  2 


!>  c 

L<  O 


05  q_*~  u 

4s*l 

“|2£So 


CO 
CD  CO 

ff 

1.13V 

CO  I 
CD  fe  Oto 

Z  CO  o  c 

CD  CL  ;  o 
CD-J^ 
^  CO 

ro  -=  cd  E 
g>.2?£ 
mlg*c 

Q  O  CL^  2 
CM  CD  <C§  I — 


a 

't 


05' yj 
CO  TO  3 
CO  NCQ 


fe_E  £ 

^  >,c I 

-c  Wo 

05.  .  CL 

op 

fSS 


2  CD  CO  c  CO  CO 

.E  >,co  E  O  k 

i=  O  CD  -£=-5  o 
CD  W  CLq  >>  ■  2- 

Q  C  CD  2-C  O 
CNLiJ0Q_C£§0 


c  W 
CO 

r- _  O)  <Z 

ll§l 

co^4a-c 
.yo«  o 

ci?js 

—  ”Qj  cd 
fe  E  co  co 
2  2  a>  $ 
o  E  ce 
_c  cd  £  o 

CD  >-2w 
CD  O  -7-3 

Q~  c 

cmC/5_<  CO 


O) 

.C 

s  c 

.2  o 
■'S  0 
35  ^ 

C  I 

CO  ^ 

£  o 


BREAK  IN  DISPLAY  AREA  (3:00  PM  -  3:30  PM) 


2  c 

o—  -e 
o>  o 


05 -D 

O 


jW 
T  CD 

5S 

oO 

=< 


I 

lgp0 

sfcflg 

feDQ  OtSO 
77  >.-D  Cl,.  CD 
O  co  c  to»E 

CM _ I  C0_§  — 


O  q; 

00  ^ 

—  „  0)  c 

Hr  O”  co  2 

cmo'Iso 


U) 

ro 

c  "co 


co 

CO 
CD 

o 

£  C  CO 

s  I® 

TO  2  .5 

i  |m  _ 

^  S-  jj  uj 

o  .§  2  w 


Track  1 

Grand 
Mesa  D/E 


Track  2 

Grand 
Mesa  F 


Track  5 

Mesa  Verde 


Track  6 

Wind  River 


SjS 

c>  CO 
1-  fe  CD 
CD  ° 

TO  o| 

0C4* 

n  to -5 

WE® 

co  CDQ. 
h-  CD  q... 
QOr^ 
CM  Cl  —  § 


to  2J 

I  8 

c  O 


<0  05 

^  O 

co~ 

.  co 

§-2 


^COTOO 

0)  fe 

0  J  0 

Q  TO  g  V)  fe 


TO  CO 

g  "O  c 

<2  S| 

£  42 

§  .2,  1 

|  O'  g5 

OQ.O 


co" . 


.—  c  <-» 
O5C0T3  E 
£  Q-C  fe 

TO  E  ro  c 

-b  o  CO  ^ 

C/3  O  CO 
_CD  E 

||2  g 

J- 

E^"<  -is 

m-TO  c 

Q-  >  0  1—  O 

E  2  c  0.58 

Z_  E ^ 

^^g-2-coo 

rsiOO  cd  —  §  — 


=  CD 

E  § 

Ww 

£  c 
_  o 


05< 

2  co 

TO  CD 

fe  05 

^  CO 

E  CD 
05  £ 

N-  Cl  CO 

°  E2 

CMbDQ 


LU  co 

co< 

o|« 
co  S  E 
2-^1 
^  ..to 


54 


W 


co  co  CD 

g  t3  C  £ 

^  SS  a 


.0  <2  N 


^  O  c  O 

I^sbiEq 
§  S  1:  w 
O  Q.  O  Q  O 


Track  7 

Wind  Star 


<0 

"D 

(/) 

0) 

c 

"D 

s 

■ 

(/) 

O 

(0 


O 


LUNCH  IN  GRAND  MESA  ABC  (12:00  PM  - 1:30  PM) 


s 


003 
CO  o 

.So 

=a= 


0)0 

1i. 


-  2  s  8* 

0-  0.2Q 

©-S 

-Ilf5 

CD  -b  ©  ©  *^ 
C0C/)-Q^Q 


CO 


CD 

C 

© 

-c 

p 


?!«§o 

^>2i 

S5£4S 


CO  C 
©  O 

8~ 

Q-  8 

?  I 

©  § 

g  8 

it 


01 

< 


S. 


0-  >c  co 

3©  O© 

§l*s 


;Q 


hPo* 


■-  'c  0 
0  30 

— 1  -© 
0  >  m  g 
gt  =5  CD  to 
j(/3  <  0 

-CO 

©-2-*=;  ©  © 

.2 

*j  O  P  t  D 

lxo(Si 

<  g-5  £  e- 

S»SlSlo 

”  ©f  ^  © 
<0^1 
co<Ollo) 


(0 

CO 

© 

o 

I- 

-O 

c 

© 

! 


l.  ©  _ 

2  .E  lu 
5  ro  (/) 

2  g  CO 

8  °  e 

■S  s  < 


Track  1 

Grand 
Mesa  D/E 


©1Z  c 

«3f| 
,11”  f 


?= 

o 


&  qE 

^  Z.  © 

®  ©0 

IIP- 

CM  0  O  _  $ 


© 

o 

c 

5 

CD 


2  § 

0-  O 


CD  g 


CO  0 
CO  § 


is  ©  © 

"5  co  ^ 
w  ©  O 
rr  ©CD 

CO 

128 

Q.1-  £, 
CLO)P> 

C.E  © 

co^  o  ,  ■ 
00^-C=i2 
co(_)C/}§ 


a 


O 

01 

0  ^6 

lo^- 

ro.ii  0  _ 
3  oOi2 

0  ©  _  C 

^SiSg 

>  ©CD CD 

|9>t: 

o-^E  © 
<  ©  O  C 
>1U  g  © 
2  g*:  o 
Q..E  q. 
CO  CO  wi;  c 

«q:dI£ 


© 

Cl 

*-  £ 
c  CO  O 

©^O 
c  0  c 
©  ©  O 
>£  © 

O  . - ci 

>  £  >, 
2  ©  © 

88* 

©£.£ 

©  o  O 


OQ 


>.©:3 

©  £  ■ 
m2  o  0 
coOO§ 


© 

C  c 

S  o 

21 
O  ^ 
o:  o 


BREAK  IN  DISPLAY  AREA  (9:30  AM  -  10:30  AM) 


2  c 

..4=0 

^©.Q 

O  c=S 
c-0^ 
-am  CD 
©  ©£ 

tsi® 

oil! 

cc  ©jS  0 

md 


©  © 

_0  © 

o  cw 


■2  ct 
zEo 
oWQ- 

©  ojo 

c.Eifc 
0  © 
©^* 
~©§ 

't;Q^0 

5  ©  o  .. -p 

<c  >  ©  it  -2 

coLUFQZ) 


LU 

CO 


~  O 
C  0 

c  c 


O  ^  ^  o 
o:  O  Q  CD 


-1  Q) 

oit 

to  &  g 
§■§-.8 

©2° 

o 

§t“ 

III 

S3:§6 


c«t: 

©  0  o 
■O  OZ 

o  ^  o 
P  .©  ©  E 
>-  >s:  o 

Sl^l 

c  o>§  c 
F  c  ©  E 

|1  o  E 

5  0)§^  | 

^  °OQCD 


©  © 

^  £  o 

.'S'  w  ©  c 

3  £  0  .2 

©  ^  2  © 
^  ©  <  2 


© 


3:  s  i 


.  o 
■  o 


Track  4 

Chasm 

Creek 


Track  5 

Mesa  Verde 


Track  6 

Wind  River 


|S 

0)0 
o  CO 
Q_  © 


n©  © 


&2 

Q> 

>  ©  °- 
o  O  -c 

^CL  ©  8 
©■©±=s 
©  c  2,^ 

F  ©I— ^  >, 
o  o  © 
E  g^§  g- 
coFll>§0 


E^© 

©^  E 

C0  o  O) 


&  0  c  g 

©  pp .2 

Eg^© 

■§  o  ©  o 

<©J  & 


©o§E 
CD  ©  ©it  >* 

commQco 


I 


<  2 
3  O-d 
o 

O  ow  . 

°=^o 

c-o  C  § 
©  o  ©  2 

s  o 

n  c  o2 

S-Bgiii 

815.1 

£8^5 


~  © 
0  Q 

ra 

2 tor  0 
<0 

22  8rS 

CDO-O’0: 

s“i2 

wf1!* 

©~  ©-C 
i-  ©  ^  o 

$  ©  — s 


U  ■■—  C\J 

-  03  .  O 

£  ©=§ 

©033OI 


Q. 

Ell 
©  +-  © 
c  =  > 
§  Q  CO 
^  t  co 

i?1 

-  E  “ 


Track  7 

Wind  Star 


ADJOURN  FOR  THE  DAY 


m 

o 

o 

CM 

CO 


Q) 

-Q 

E 

0) 

> 

o 


> 

05 

"D 

m 

Q) 

G 

"D 

0) 

£ 

■ 

if) 

O 

05 


o 


CL 

in 


Cl¬ 

io 


|E 

>-  CD 
—  .CD 


£2 


s-c/ 

5c 


oi*:.. 

!q  +-  c^-m  c 

illii 


jz 

Q  25  CD 

Eo^~q_§q: 


2  CD 
0  E 


"EC 

§>. 


jjj  o  u: 

j)  (0 

00  LU  >  CD 

«g  Pts  § 

0l"X 

Q^o©c2 

ooOCOCLQcO 

<0  3** 

W)  c 
0)  o 

s“ 
a-  8 

?  I 

CD  § 

II 


S"Q 

Q  0 
+-  £= 
—  0 

1 1 

^CO 

o  ^ 
0  0 
x:  c 


“  o 


o 

o--  ^ 

£H*  0 

t-  cd  « 

C_)2  ^  0 

co  -E  □  CL 


'■a 

uP  0°^ 

0  0.SZ  C 

c  £  >5 

—  O  CD  O) 

$-2*q> 

$< 

0"0  0«q: 

J-  C=  0^ 

CL  CD  ^  TO 
0  0^  £ 
«ocn5? 

O.O.0Q  §_ 

5l??SS 

colu5cq§0 


</>  0 
£  ■*= 
8  ® 
s  -  = 

ft*  C  (0 

Si® 
s  si 

1 

o  ^  i 


Track  1 

Grand 
Mesa  D/E 


0 

>,  -£ 
C'  0  0 

2  SH 

=  I  - 

5  5 


c n 

-  Eg 

0  TO^ 

° 

«  >N  «5 

0  CD 
n”§8 

>3Qr 

mroo.'t 
Q  C-Q5  ° 
co<<§Z 


C  o 

II 

■§!■ 
CO  o 

co 


LU 


:  «5 


2io«E 

SSfiiio 


co  75 
0  3: 

£c^r 
-£§  .1 
>.N  0  2 
0I  0^  Q- 
2i;  o 
oOO§Q 


0ro  >, 
0~  o 
-*3  LU  O 
o3^  <  c 

0^  co 

1I4I 

0-j=  0  ?  E 
=  c  c.S>  2 
_c=  2  cNto 
°  0  2iU  E 

c  CL>  0  o 

g£iu25 
0-d  <g  |  w 

030  2.C  £ 

CD  E  0  ^  , .  0 
a”?  0  05  >> 
CD^_JCO§CO 


(/) 

C 

o 

55 

c 

0 

UJ 

I  c 

S  8 


BREAK  IN  DISPLAY  AREA  (3:00  PM  -  3:30  PM) 


0^sE 

si! 

— -C  is 
^  00 

olSg 

0  0  ’■£ 

x:  03  0  0 

csjSl^e- 


"c=9r 

0< 

15 


0  CO 


cxc  g  < 

I^I.? 

llfll 

o  id  0  ro  - 
2  o  Eq_S 
Q.-Q  8<*  ^ 
cd  n  c 
03=  tf  „ 


,  J  S 

■  >y* 


"CB°  0  = 

CM  Q  O  0  «=  .  t 

o  o  ^  >o«|  o 
cd<2coO§0 


0  o 
go 

s  = 

3  o 

2o 

0  — 

£  0 

o  Q. 


£  C 

E  ° 

jo  0 
-c= 

<P 

I|“ 

=  ^  O 
Q_^  O 

O  0<J 

O  c  0 
^  0^ 
50O 
o  ^.o  >, 

cdZ5  To 

■E02o. 
CO -o-o 7:  £ 
O.E  ci  o 
coll  to§Q 


0 

01 


0 


!  | 
=5  5 

o  0 
5  o 
O  ^ 


1.1  SS 

O  CDs^  Q. 
>.V  Q_  CL  .  £ 

O  co5  o 
co  — <§Q 


c 

S.2 

0  +i 

S'  2 
S  o 

w  Q  E- 
|  So 
2  o  ■" 

Q.  ^  0 

<C  S  I 


C  o 
■S'CD 

j£  o 

o  &■ 

CO  o 

co 


0LO  TO  E 

_,.  2  0^  £ 


00 

—  Q. 
ZJ  O 
£=  0 
0  CL 

£  0 
P  0 


£-£ 

£0 

— -Q 

0  O 

w  E 

0  CL 


ct 

^■EE'S 

=  d0)5 

si^oS 

coO  0  cl§ 


LU 

CO 


oj  S;  0  0 

^  ^  Q  -a 

iJ  o 

o:  o  q  o 


Track  2 
Grand 
Mesa  F 


Track  3 

Highlands 


Track  4 

Chasm 

Creek 


.2  O 

"0.2  — 

0"O 

m52 

.Emo.  | 

2g?0 
C  ^  0O 

5  r  C  to 

P  c-  - 


o 


.  0  V) 
0  ££ 


’5  c rv- 

ujt  0QC 
■p?0  0  . 

2o_  0^r 

n  0^ 

.E2<$ 

JsM 

CO  CL  0Q_^ 


Q. 

si! 

0  t;  0 
c  =  > 
S  Q  CO 
b  it  w 

M? 

Is-? 


Track  7 

Wind  Star 


in 

o 

o 

CM 


Q) 

-Q 

E 

Q) 

> 

O 


<0 

TJ 

0) 


( /> 

O 

(0 


(0 

o 


o 

.0) 


O 


LUNCH  IN  GRAND  MESA  ABC  (12: 

PM  -1: 

PM) 

-Q  *J 

Ot  o 

j  ag 

050-  . 

£=  =5Q 

To-i 

CQI  2 

>-  &  o 
.©  a3& 
b"o^ 
33  cd  55 


O 


O) 


1=1 
-I® 


I, 


£  .2 

O  ‘ 


a“- 
gi  w 


CO 


Track  1 

Grand 
Mesa  D/E 


CD  "O 
— I  CD 
2  C 

£  'to 

o  to 


Q.w-» 

i?i* 

—  T3  CDS 

O  CD  >  C 

Oj  >  CD  CL,. 

CO  O  CD  S=«^ 


Q 

°3 

a: 


X  □ 
<  CO 
£  -5C 

05  ^ 

<o| 
-£w© 
^  oO 

CM^  O  .. 
00^  =5i 

'vi-OcoS 


CO  c 

L.  0 

Q-  o 


o  o 

S£  § 

|  o  5> 

pcojo 

52  © 

03  CD  ^ 

Sot3: 

co.E  jo's 

CO  -£  CD  3 
a)  Co_ I  CD 

oor  ^o. 

sills 


>0 

CL  >, 

II 

col 
o  ^  _ 
0'OLIJ 

c  Eco 

o  . 

c  O  Q? 
CO  ©& 

^5! 

CO  CD  CD  . 
CQJ=-^^ 
'd'l— 


a 


E 

2  LU 

E:  cc 

CD  I— 

.52  § 

CD  S 
Q.  CD 
CL  -C 
<CDI_. 

o.^> 

x©g 

<  £  § 

O  05^ 

too-J 
cd  o®-  o 

rn  05  C+-; 
C  ©-C  2 
'E  ro  o  o 
^  c=  i=  *  0- 

goco^:  o 

0_  CD  §  O 


©  LU 
>  CO 

rr  *• 

LL  05  C 

III 

3c* 

CD  OS 


LU  ‘ 


:  CD 


CO  *«. 
CO  o  o 

0  o  .1 

t 

Q  O  s_  (/) 

tOl£§ 


”  CD  Z 

-nil 

pl«l 

og|o:8 

O5'-=LJi0  c 

1-^.se 

“ujOmSO 


XN  cO 
CD=  ©  C 

>,0^  p 

£*o|5 

I  §0 


■c 

.O) 

3: 


BREAK  IN  DISPLAY  AREA  (9:30  AM  -  10:30  AM) 


<f-i 

ill 

1*3 

CD  — 

^<o§ 


LO  C 
<  CD 

3=0 


©  © 

S-  _£  O 

■’s  (o  2  c 
=  E  ».o 

S  ?L  J 
^  s-  ©  ir 
^  ©  <  2 
>C  ~ 5  p. 

Siijo 

IShU 


f-o  C5 

i§  = 

3  0  c 

Q<  2 

4--o  ,S 

00  n 
£!>  ? 
f|i!e 
sfll^ 


E 

© 

to 

>. 

CO 


CD 


—  LU 
©._<  >; 

=3  ©“-§ 

s_  C  ©  o 

O  P  0)0 
m-  5  m  JZ 
—  o  2  o 

^  c<0^ 

OLiJ^o 

5P«Sl 


(0  TO 
C/>  J_  c 

c  ©  c 

«  I  i 

I?SJ 

>c  T3  Q.  t: 

Uj  c  o  E 
—  co  2  o 
1“  t  fr 

BsI5 


Track  4 

Chasm 

Creek 


Track  5 

Mesa  Verde 


Track  6 

Wind  River 


O  CD 

QlW 

>p 
o4- 
-Q  0 

t!  o 

is  28 

og-I^ 

O  _E  0  O 
g  I  COO 

S'oEq: 

o  o 

i-  c  o  3 

°-  EJS  TO 
©  o^Q- 
Is-  O  0-0  . 

DQOwc^ 
"3-0<  CD  § 


0 


©  ^ 
o  .> 

©  .2 
i  1 
—  _  © 
05^  0 

0 

■[:2® 
©o^ 
c  ©  © 

cS^ 

0°^ 

E  05.S 
O 

n- to  2  ..o 
m  ,>--d  ^  — 1 

'vfCOI— §_1 


5  © 


<^«s 


Track  7 

Wind  Star 


CD 

CO 


>> 

CD 

O 

O 


O 

0 


O 

o 


<D-o 

5  c  “ 

E ro 
o 

CD  CO  7  ■ 

£Q  O 

•">.  Cg  = 
£J  CD  2  - 

i^sic 
E  JS  2  <8 -2 

c  o£  <S<o 

TO  <-  -W  w  CO 

<NiS-§-g^g 
□roc 2  x. 2 
■^t  Q  LLI  Q_  §  Q_ 


CM  C 

o  TO 

LLJ 

gS 

1 1 


Track  2 

Grand 
Mesa  F 


CO 


g  ^ 


O 


TO  2 
£-2 

■■=-oJW 
ScCc 
■■=  TO  TO  o 

Sto00^-^ 


Track  3 

Highlands 


Track  4 

Chasm 

Creek 


o 


CD  — ■* 

TO  TO 

-  |S 
■o  <Q 

CO  CO 

I  §1 


■^LO  TO  c 
CO  —  .TO  — 

CD  CD  ?5  Ol 

•es^iS 

q^S£ 

■—  -2  CO 

»s!^r 

S?£=4fil 


CD 


-  CO 

>s 

.  o  £ 

£“  s  E 

fc  S  O  o 
2Sas 
5  -o  o  2 
^  c  £  o 
■C  <  t  Q. 
>2>  jj  o  o 
a:  s  z  o 


Track  5 

Mesa  Verde 


CD  C 
o  CO 


O 

.£=_ 

~LU 

^C/) 

05^- 

.E  <5 

p 

s§ 

CD  C 

II 

CD$i 
O  ox: 
^-c/)Q 

00  I 

C  CD  = 

.2  S  E 

l*o  § 
file? 

— .  To  ^  O 

!“  t  & 


Track  6 

Wind  River 


Track  7 

Wind  Star 


Thank  you  for  displaying  at  the 
5th  Annual  CMMI  Technology 
Conference  &  User  Group 


Borland 

Carnegie  Mellon  Software  Engineering  Institute 

Cognence,  Inc. 

Galorath,  Inc. 

Integrated  System  Diagnostics,  Inc. 

Kamo  Consultancy 
Kasse  Initiatives,  LLC 
Level  Five  Solutions 
MKS,  Inc. 

Pragma  Systems  Corporation 
Q-Labs,  Inc. 

Quality  Improvement  Consultants,  Inc. 
Quantitative  Software  Management 

Raytheon 

Raytheon  Networking  Centric  Systems 
Schweitzer  Engineering  Labs 
The  Center  for  Systems  Managment 
The  David  Consulting  Group 


STJtfcWTU TUKOUGLI  INDUSIM  ii  TliiClINOLOO) 

2111  Wilson  Boulevard 
Suite  400 

Arlington,  VA  22201 
www.ndia.org 


5th  Annual  CM  Ml®  Technology  Conference  &  User  Group 
Hyatt  Regency  Tech  Center  ♦  Denver,  CO 


SYSTEMS  AND 
SOFTWARE 
CONSORTIUM 

BUILDING  BETTER 
SOLUTIONS  TOGETHER 


CMMI®  and  Stakeholder 
Involvement  ✓  ®, 

Worth  the  effort?  CMMI 


Date:  November  14,  2005 
Presented  By:  Jim  Armstrong 


Systems  and  Software  Consortium  /  2214  Rock  Hill  Road,  Herndon,  VA  20170-4227  www.systemsandsoftware.org 

Phone:  (703)742-8877  /  FAX:  (703)742-7200 


Agenda 


•  CMMI  Stakeholder  Content  -  What  it  is 
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•  Issues  -  What’s  the  problem 

•  Conclusions  -  So  what  now 
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CMMI  Stakeholder  Content 
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CMMI  Terms 


•  Stakeholder 

-  A  group  or  individual  that  is  affected  by  or  in  some 
way  accountable  for  the  outcome  of  an  undertaking 

•  Relevant  stakeholder 

-  A  stakeholder  that  is  identified  for  involvement  in 
specified  activities  and  is  included  in  an  appropriate 
plan 


11/23/2005 


CMMI 


CMMI  Comprehensive  Stakeholder  Tasks 


GP  2.7  Identify  and  Involve  Relevant  Stakeholders 

-  Identify  and  involve  the  relevant  stakeholders  as  planned. 


PP  SP  2.6  Plan  Stakeholder  Involvement 

-  Plan  the  involvement  of  identified  stakeholders. 


PMC  SP  1 .5  Monitor  Stakeholder  Involvement 


-  Monitor  stakeholder  involvement  against  the  project  plan. 


IPM  SP  2.1  Manage  Stakeholder  Involvement 

-  Manage  the  involvement  of  the  relevant  stakeholders  in  the 
project. 
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CMMI  Stakeholder  Involvements  in  SPs 


Explicit  in  Almost  all  PAs,  particularly: 

•  Obtain  plan  commitment 

•  Identify,  negotiate,  and  track  critical 
dependencies 

•  Resolve  issues 

•  Develop  requirements 

•  Measure  and  analyze 

•  Establish  teams 

Implicit  -  GP  2.7 
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CMMI  Stakeholder  Sources 
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CMMI  Source  Models  for  Stakeholders 


SW-CMM  EIA/IS  731 

IPD-CMM 
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SW-CMM 


•  Review  by,  participation  of.. 

-  Other  affected  groups 

-  Other  engineering  groups 

-  Software  related  groups 

-  Management 

-  Customer 

•  Identification  of... 

-  Statement  of  work  covers 

-  ...  customer  and  end  user 

•  Monitoring  ?? 
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EIA  731 


•  Stakeholders  include 

-  Customer/users,  developers,  producers,  testers,  suppliers, 
marketers,  maintainers,  disposers 

-  Others  who  may  be  affected  by,  or  may  affect,  the  system  or 
product 

•  Stakeholders  are: 

-  Involved  in  requirements 

•  engaging  all  stakeholders  in  an  ongoing  dialogue 

-  Review  plans 

-  Coordinate  disciplines 

•  Identify?? 

•  Monitor?? 


CM  Ml 
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•  A  stakeholder  is  defined  as  one  or  a  group  who  has/have  a 
direct  impact  on  or  from  the  product  or  its  production 
process 

•  References  to 

-  Identified 

-  Communicate  with 

-  Collaborate  with  on  vision 

•  Identify??  Note  under  leadership 

•  Monitor?? 

•  Not  released  -  no  use  and  appraisal  experience. 

CM  Ml 
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Additional  Notes 


History  of  Integrated  Product  Development 

•  Stakeholder  involvement  has  significant  impact 

-  Performance 

-  Cost 

-  Schedule 

•  Good  involvement  not  natural 
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Process  Stakeholder  Approaches 
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Range  of  Options 


Assume  it  will  work 

Let  integrative  tasks  push  involvement 
Emphasize  identification 
Monitor  participation 


-  Quantity 

-  Quality 

Address  impact 

-  Identify  issues 

-  Take  actions  to  improve  involvement 
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Participation  Planning  in  Processes 


How  handled 

•  Pre-defined 

-  Roles  and  responsibilities 

-  Defined  participant  lists 

•  Definition  tasks 

-  Stakeholder  matrices 

-  Lists  in  plans,  agendas 

Results 

•  Strong  on  who  is  a  stakeholder 

•  Weak  on  involvement  specifics 
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Monitor  and  Manage  in  Processes 


•  Attendance  taken,  but  not  much  participation  tracked 

•  Meeting  canceled  if  missing  or  not  prepared 

•  Not  much  in  the  way  of  raising  as  an  issue  or  taking 
corrective  action 

•  Could  be  included  as  a  measurement 
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Observations 


•  General  response  -  “We  did  this  anyway” 

•  Some  -  “Made  us  think  about  it” 

•  Many  changes  due  more  to  basic  activity  task  than 
stakeholder  practice 

-  Requirements  review  now  used 

-  Integrated  plan  review  brings  stakeholders  together 


•  Much  on  identification,  little  on  managing 
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The  Issues 
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Increased  emphasis  on  monitoring? 


•  Can  a  program  manager  answer: 

-  How  is  stakeholder  involvement  working  on  your 
program? 

•  Happening? 

•  Impact? 

-  What  is  the  basis  of  your  conclusion? 

•  What  is  the  best  balance  between 

-  The  value  of  getting  a  better  answer 

-  The  effort  needed  to  get  the  answer 
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CMMI  Model  Options 


•  Eliminate,  merge  or  reduce  requirements 

-  Delete  GP  and  let  PP,  PMC  and  IPM  suffice 

-  Reduce  monitoring  language 

•  Leave  requirements  alone 

-  Maintain  current  status 

-  Change  the  way  appraisals  look  at  GPs 

•  Add  requirements 

-  Be  more  specific  on  assurance  that  planned  involvement  is  as 
anticipated 

CMMI 
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Conclusions 
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Conclusions  -  Usage 


•  Quite  a  range  of  options  are  available 

•  Minimum  option  is  inviting 

•  Greatest  weakness  is  on  managing 
involvement 

•  I  PD  history  shows  impact  of  stakeholder 
involvement 

•  You  must  decide! 
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Conclusions  -  Model 


•  Need  to  consider: 

-  Is  everybody  really  tuned  in  to  real  stakeholder 
involvement? 

-  Can  appraisal  issues  be  resolved? 

-  Does  the  Generic  Practice  add  significant 
value? 

•  Better  data  would  help  decision  on  content 
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Who  is  GTRI? 


•  Unit  of  the  Georgia  Institute  of  Technology 

•  1200+ employees 

•  Wide  variety  of  products 

•  Customers  include  federal,  state,  and  industry 

•  Projects  range  greatly  in  size  and  duration 

•  More  lnfo:http://www.atri.aatech.edu/ 


Current  Status 


•  Assessed  CMM  level  3 

•  Performed  gap  analysis  between  CMM  and  CMMI 

•  Updating  processes 

•  Implementing  the  new  processes 

•  Not  assessed  under  CMMI 


Outline 


•  CMMI  and  peer  reviews 

•  Purpose  of  peer  review 

•  Formalize  the  peer  review  process 

•  Plan  peer  reviews 

•  General  example  of  the  execution  of  a  peer  review 

•  Secondary  benefits  of  peer  reviews 
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CMMI  Verification  Process  Area 

Specific  Practices 


SGI  -  Prepare  for  Verification 

SP  1.1-1  -  Select  Work  Products  for  Verification 
SP  1.2-2  -  Establish  the  Verification  Environment 
SP  1.3-3  -  Establish  Verification  Procedures  and  Criteria 
SG  2  -  Perform  Peer  Reviews 

SP  2.1-1  Prepare  for  Peer  Reviews 
SP  2.2-1  Conduct  Peer  Reviews 
SP  2.3-2  Analyze  Peer  Review  Data 
SG  3  -  Verify  Selected  Work  Products 
SP  3.1-1  Perform  Verification 

SP  3.2-2  Analyze  Verification  Results  and  Identify  Corrective  Action 
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What  is  a  Peer  Review? 


“The  review  of  work  products  performed 
by  peers  during  development  of  the 
work  products  to  identify  defects  for 
removal.” 


-  CMMI  Guidelines  for  Process  Integration  and  Product  Improvement 

(Addison  Wesley,  2003,  page  622) 


What  is  Verification? 


“Confirmation  that  work  products  properly 
reflect  the  requirements  specified  for 
them.” 


-  CMMI  Guidelines  for  Process  Integration  and  Product  Improvement 

(Addison  Wesley,  2003,  page  631) 
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Purpose 


•  Verify  the  work  product  meets  requirements 


•  Identify  defects  or  problems  early  in  the  life-cycle 


•  Gain  confidence  in  work  products 

•  Reduce  risk 
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An  Informal  Peer  Review 


“Does  this  seem  right  to  you?” 
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An  Inappropriate  Peer  Reviewer 


“Farmer  Bob,  does  this  seem  right  to  you?” 
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Why  Do  We  Need  Formalized  Peer 

Review  Processes? 

CMMI  requires  it! 

A  formalized  process  helps  ensure: 

•  Peer  reviews  are  taking  place 

•  The  right  products  are  being  peer  reviewed  at  appropriate  times 

•  Adequate  resources  are  planned  and  allocated  for  peer  reviews 

•  The  right  reviewers  are  being  selected 

•  The  reviewers  are  prepared  adequately 

•  Defects  are  being  recorded 

•  Defects  are  tracked  to  closure 


Establishing  a  Peer  Review  Process 


Establish  procedures  for 
peer  reviews 

Establish  “ground  rules” 
for  peer  reviews 

Provide  guidance  in  what 
&  when  to  peer  review 
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Document  the  Peer  Review  Process 

•  Types  of  reviews 

•  What  to  review  in  each  phase 

•  Planning 

•  Conducting 

•  Closing 
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Peer  Review  Types 


Desk  Check 

•  Single  producer  and  single  reviewer 

•  Cheapest,  least  effective  review 

Round  Robin 

•  Single  producer  and  at  least  two 
reviewers 

•  Reviewers  examine  work  product 
sequentially 

•  A  single  defect  log  is  used 

•  Moderator  verifies  defects  are 
corrected 


Structured  Walkthrough 

•  At  least  two  reviewers,  a  Moderator, 
and  a  Recorder 

•  All  participants  meet  after  reviewers 
have  prepared 

•  More  expensive  and  effective  than  a 
Round-Robin 

Formal  Inspection 

•  Roles  and  format  similar  to 
Structured  Walkthrough 

•  Outside  experts  participate 

•  Advanced  preparation  is  extensive 
and  required 

•  Most  expensive  and  effective 
review  type 
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What  to  Review 


Requirements 

Design 

Implementation 

•  Critical  components 

•  Complex  components 

•  New  employee’s  work 

•  New  technology  or  platform 
Test  Plans 
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Plan  Peer  Reviews 


•  Determine  what  will  be  peer  reviewed 

•  Determine  when  it  will  be  peer  reviewed 

•  Provide  adequate  budget  for  peer  reviews 

•  Plan  for  critical  reviewers 

•  Plan  for  appropriate  facilities 

Georgia  I 
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Applying  the  Process 
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Prepare  for  Peer  Reviews 


•  Choose  reviewers 

•  Schedule  meeting 

•  Prepare  review  and  reference 
materials 
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Choosing  Reviewers 


•  Knowledgeable  and 
trained 

•  Some  project-independent 
reviewers  are  desirable 

•  Non-management,  unless  special  circumstances 
require  a  manager’s  participation 

•  Committed  to  adequately  prepare 
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Scheduling  the  Meeting 


Allow  the  reviewers  adequate  time  to  prepare 
and  turn  in  defect  logs 

Define  clear  objectives  regarding  the  amount 
of  time  (min/max)  for  the  review  preparation 

Limit  meeting  time  to  two  hours 

Ideally  choose  a  location  with  a  networked 
computer,  overhead  projector,  and  access  to 
configuration  management  system 
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Review  and  Reference  Materials 


Review  materials  must  be  under 
version  control 

Provide  controlled  defect  logs  to 
reviewers 


Identify  location  and  version  of  all 
review  materials 


Provide  reference  materials 
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Preceding  the  Peer  Review 


•  Verify  producer  has  distributed 
product 

•  Verify  that  reviewers  are 
prepared 

•  Tabulate  all  the  defects  into  a 
summary  log 
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Conducting  the  Meeting 


•  Walk  through  the  work  product  in  its  entirety;  don’t  just 
look  at  the  tabulated  defects 

•  Ideally  -  use  a  projector  so  that  everyone  can  see  how 
defects  are  recorded 

•  Gain  consensus  during  the  review  of  the  type,  severity 
and  disposition  of  each  defect 

•  Identify,  but  don’t  try  to  fix  the  defects 

•  Determine  if  re-review  is  necessary 
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Closing  the  Peer  Review 

•  Put  peer  reviews  on  the  list  of  project  deliverables 
so  that  closing  them  won’t  fall  through  the  cracks 

•  Close  out  defects  within  30  days  or  write  a  change 
request 


•  Require  project  director  and 
quality  engineer  signature  tc 
close  the  review 


*  Re-review  if  necessary 
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Secondary  Benefits 


*  Create  mini-milestones  for  work  products 

•  Jump-start  team  communication 

*  Product  quality  increases  when  the  author  knows  it 
will  be  reviewed 

•  Create  an  esprit  de  corps  within  the  project  team  - 
everyone  has  to  be  reviewed  and  act  as  a  reviewer 


More  Secondary  Benefits 


•  Leverage  team  member  skills 

•  Teach  junior  engineers  “It’s  OK  to  criticize  senior 
people’s  work” 

•  Exposes  junior  engineers  to  direct  tutelage  from 
experts 

•  Expose  reviewers  from  outside  the  project  team  to 
new  ideas,  and  vice-versa 
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Verification  In  CMMI  Using  Peer  Reviews 

AUTHORS:  JEANNE  BALSAM,  JEAN  SWANK  LEE  SHEINER,  AND  MARK  PELLEGRINI 


INTRODUCTION 


Georgia  Tech  Research  Institute  (GTRI)  is  the  nonprofit  applied  research  arm  of  the  Georgia 
Institute  of  Technology  in  Atlanta,  GA.  The  Electronic  Systems  Laboratory  (ELSYS)  of  GTRI 
achieved  a  CMM  Level  3  rating  in  June  of  2003.  ELSYS  employs  150  engineers  and  scientists 
working  predominately  on  DoD  related  competitively  bid  contracts.  Over  the  last  30  years,  ELSYS 
researchers  have  established  national  reputations  in  areas  such  as:  monopulse  countermeasures, 
advanced  radar  warning  receiver  design,  survivability,  simulation  models  and  analysis,  and  Electronic 
Countermeasures  (ECM)  technique  development.  GTRI/ELSYS  core  competencies  include 
software  and  systems  engineering  for  electronic  warfare  and  avionics  systems,  reliability  and 
maintainability  upgrades,  technology  insertion,  obsolescence  programs,  threat  analysis,  and  mission 
critical  software. 

This  paper  describes  the  Peer  Review  process  utilized  by  GTRI/ELSYS.  The  procedures  were 
developed  originally  to  meet  CMM  for  Software  requirements.  Over  the  last  three  years  those 
processes  have  been  extended  to  cover  all  systems  components.  These  reviews  can  also  aid  in  the 
validation  of  the  product,  if  planned  effectively  and  with  the  correct  reviewers.  Key  features  of 
effective  peer  reviews  include: 

•  Selection  of  proper  type  of  peer  review  for  each  work  product  and  phase  of  development. 

•  Selection  of  peer  reviewers  who  have  the  required  knowledge  and  independence 

•  Detection  of  defects  early  in  the  lifecycle 

•  Reduction  of  risk  of  latent  defects 

•  Documentation  of  defects  and  tracking  them  to  resolution 

•  Utilization  of  supporting  tools  and  environments  for  effective  reviews 

•  Configuration  management  of  review  and  reference  material 

Other  benefits  of  peer  reviews  include: 

•  Consistency  in  the  use  of  engineering  standards  and  practices 

•  “Cross-pollination”  of  ideas  and  methods  across  the  organization 

If  your  organization  is  considering  employing  a  peer  review  process  you  should  find  this  information 
helpful  in  getting  started.  If  you  are  already  regularly  conducting  peer  reviews  you  might  find  some 
of  the  topics  discussed  helpful  in  fine-tuning  your  process. 
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THE  CMMI  VERIFICATION  PROCESS  AREA 


Peer  reviews  were  a  “Key  Process  Area”  in  the  Capability  Maturity  Model  (CMM)  for  Software. 
Under  CMMI  peer  reviews  are  defined  as  a  specific  goal  in  the  CMMI  Verification  process  area. 
Essentially,  the  CMMI  guidelines  correctly  classify  peer  review  as  a  method  of  verification.  Other 
methods  of  verification  include,  but  are  not  limited  to,  audits,  analysis,  simulations,  testing,  and 
demonstration.  Peer  reviews  (inspections,  structured  walkthroughs)  can  be  one  of  the  most  cost- 
effective  and  productive  tools  you  can  use  to  create  robust  products  on  schedule  and  within  budget. 
That  is  their  theoretical  potential.  Without  the  other  components  (specific  goals)  of  Verification, 
Prepare  for  Verification  and  Verify  Selected  Work  Products,  peer  reviews  lose  value.  For  peer 
reviews  to  be  effective  they  must  be  planned  to  take  place  throughout  the  lifecycle  of  the  product. 
Additionally,  verification  procedures  and  criteria  must  be  in  place  along  with  an  environment  that 
facilitates  the  peer  review  process. 

There  are  several  important  objectives  that  peer  reviews  can  accomplish. 

Identify  Defects  Early 

Early  identification  of  defects  is  extremely  important,  and  the  ability  of  peer  reviews  to  ferret  out 
defects  before  they  become  expensive  to  fix  is  probably  their  single  greatest  benefit.  A  defect  in 
requirements  may  only  take  a  few  hours  to  address  and  correct.  If  that  same  defect  makes  it  from 
requirements  into  design  and  implementation,  and  is  not  caught  until  testing,  it  might  take  hundreds 
of  hours  or  more  to  fix,  as  well  as  make  it  impossible  to  deliver  the  product  on  schedule  and  within 
budget.  Early  detection  pays  big  dividends. 

Verification 

Peer  reviews,  or  other  verifications  must  be  scheduled  throughout  the  lifecycle  of  the  system. 
Verification  is  the  act  of  verifying  that  a  work  product  is  being  built  which  implements  the 
requirements  of  the  system.  As  the  Reviewers  look  at  a  work  product,  they  should  make  sure  that  it 
is  consistent  with  the  work  products  produced  earlier  in  the  life-cycle.  For  example,  the  Reviewers 
need  to  verify  that  the  design  is  not  only  feasible  to  implement,  but  implements  the  requirements 
placed  on  the  module.  Likewise,  a  peer  review  of  a  module’s  implementation  should  be  verification 
that  it  accurately  implements  its  design.  Thus  in  all  phases  of  the  lifecycle  the  work  products  are 
verified  against  previous  work  products  and  against  the  system  requirements. 

Gain  Confidence  in  Work  Products 

For  project  directors  and  technical  leads,  peer  reviews  increase  their  confidence  that  the  team  is 
making  progress  towards  product  completion.  Setting  the  correct  criteria  for  peer  reviews  and 
making  a  peer  review  the  gate  that  a  work  product  must  pass  prior  to  being  considered  complete 
provides  the  project  director  and  technical  leads  with  concrete  information  on  which  to  base  their 
plans.  By  peer  reviewing  work  products  one  reduces  the  probability  that  some  critical  link  is  weak  in 
the  chain  from  requirements  to  a  finished  product. 
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Reduce  Risk 

By  identifying  defects  early  in  the  product  life-cycle,  peer  reviews  help  reduce  the  risk  of  late 
delivery,  a  bad  product,  or  an  overrun.  This  benefit  is  achieved  by  performing  peer  reviews  during 
each  phase  of  a  product’s  development.  Early  identification  of  risk  areas  allows  for  development  of 
effective  and  timely  project  risk-mitigation  strategies. 


PLANNING  PEER  REVIEWS 


Peer  reviews  don’t  just  happen  by  themselves.  They  must  be  a  part  of  project  planning,  otherwise 
sufficient  resources  will  not  be  allocated  to  support  them.  Not  only  must  time  and  budget  be 
considered  and  planned,  but  critical  personnel  resources  may  need  to  be  secured  through  formal 
commitments.  At  a  minimum,  peer  reviews  of  important  work  products  need  to  be  scheduled  as 
tasks  on  the  project  schedule.  In  ELSYS,  peer  reviews  are  required  for  requirements,  design,  and 
test  procedures,  so  these  peer  reviews  are  part  of  the  normally  scheduled  project  activities.  Peer 
reviews  of  selected  code  units  are  also  required  and  scheduled.  Placing  these  peer  reviews  on  the 
project  schedule  not  only  assures  that  they  are  planned,  conducted  and  closed,  but  also  assures  that 
they  are  specifically  funded  when  budgeting  for  product  development. 

All  peer  reviews  can  not  be  pre-planned  at  the  being  of  a  project.  As  with  all  major  project  tasks 
management  reserve  should  be  set  aside  for  follow-up  peer  reviews  and  additional  peer  reviews, 
which  are  scheduled  for  new  or  modified  components. 


WHAT  AND  WHEN  TO  PEER  REVIEW 


It  is  essential  to  conduct  peer  reviews  at  appropriate  points  in  product  development.  Finding 
defects  early  in  the  product  development  process  (i.e.  requirements  and  design)  provides  the  greatest 
return  on  investment.  The  ELSYS  process  requires  that  a  Structured  Walkthrough  is  conducted  on 
all  documentation  (e.g.,  Word  Document,  slides,  video,  etc.)  of  requirements  and  designs.  Test 
plans  and  test  descriptions  must  be  peer  reviewed  with  at  least  a  Desk  Check,  and  in  practice  it  is 
preferred  that  they  be  reviewed  with  a  Structured  Walkthrough  that  includes  key  designers  and 
implementers  of  the  product  being  tested.  Deciding  what  to  review  when  peer  reviewing 
implementations  is  less  straightforward  when  considering  systems.  Software  development  and 
hardware  development  have  specific  implementation  details  that  warrant  discussion.  The  following 
paragraphs  provide  some  guidance  on  deciding  the  type  of  peer  reviews  that  should  be  scheduled 
and  major  concerns  specific  to  each  type  of  development. 

Software  Implementations 

In  an  ideal  world,  all  code  units  would  be  peer  reviewed  with  at  least  a  Desk  Check.  We  do  have 
some  projects  where  that  is  a  requirement,  particularly  ones  that  are  mission-critical.  But  sometimes 
the  amount  or  severity  of  the  defects  found  may  not  justify  the  expense  to  find  them  as  compared  to 
discovering  them  in  unit  testing  or  acceptance  testing.  In  these  cases  a  judicious  use  of  limited  peer 
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reviewing  resources  is  warranted.  The  goal  is  to  select  a  sub-set  of  code  units  for  peer  review.  The 
following  paragraphs  provide  guidance  in  that  selection. 

If  the  project  team  is  using  some  new  language  or  other  new  technology,  a  Structured  Walkthrough 
of  one  of  the  first  units  produced  is  recommended.  This  will  give  the  entire  project  team  a  chance 
to  share  their  insights  into  the  new  tool  using  a  real-world  example.  All  who  participate  can  benefit, 
not  just  the  Producer  of  the  product  being  reviewed. 

Code  that  is  written  by  someone  who  is  inexperienced  and / or  new  to  the  project  team  is  a  good 
candidate  for  peer  review.  In  these  cases  it  is  especially  important  to  conduct  some  type  of  peer 
review  as  soon  as  possible,  preferably  with  their  first  code  unit.  If  there  are  problems  with  coding 
standards,  implementation  approaches,  or  other  aspects,  it  is  important  to  find  and  correct  these 
problems  before  they  have  been  replicated  in  a  large  number  of  coded  units.  As  always,  early 
detection  brings  the  most  benefit. 

Critical  units  are  also  good  candidates  for  peer  reviews.  If  a  unit  is  essential  to  the  performance  of 
the  product  it  should  be  peer  reviewed.  For  example,  a  base  class  from  which  numerous  other  units 
are  derived  should  usually  be  examined.  A  defect  in  this  class  might  cost  significant  debugging  time 
for  many  developers,  magnifying  the  cost  of  the  defect.  Code  units  that  are  complicated  and  hence 
prone  to  errors,  such  as  implementations  of  complex  algorithms,  are  also  good  candidates  for 
review.  If  the  software  is  time-critical,  units  that  are  expected  to  be  executed  the  most  during  run¬ 
time  should  be  reviewed  with  an  eye  towards  efficient  implementation. 

Hardware  Implementations 

By  hardware  implementations  we  are  referring  to  the  detailed  construction  designs  from  which 
functioning  hardware  can  be  fabricated.  This  includes  circuit  board  schematics,  fabrication 
drawings,  etc.  When  deciding  what  hardware  implementations  should  be  peer  reviewed,  if  the 
schedule  or  resources  permit  only  a  sampling  of  some  work  products,  the  same  types  of  criteria  that 
apply  to  software  apply  here  as  well.  Implementations  using  new  technologies,  created  by 
inexperienced  producers,  or  ones  that  are  critical  units  are  good  candidates  for  peer  reviews. 

For  hardware  items  a  further  consideration  is  critical  path  and  schedule.  A  long-lead  item  would  be 
a  candidate  for  peer  review  if  it  is  in  the  critical-path.  The  cost  of  peer  reviewing  the 
implementation  might  be  significantly  more  than  the  cost  of  the  item  itself,  but  a  peer  review  would 
still  be  justified  if  a  schedule  slip  would  be  extremely  costly  or  multiple  items  are  being  produced. 


REASONS  FOR  FORMALIZING  THE  VERIFICATION  PROCESS 


If  you  have  never  participated  in  a  peer  review,  you  may  be  wondering  exactly  what  a  peer  review  is. 
Actually,  it  is  likely  that  you  have  already  participated  in  many  peer  reviews  without  even  knowing  it. 
Walking  down  the  hall  and  asking  a  co-worker  to  look  at  something  you  have  produced  is  essentially 
a  peer  review,  boiled  down  to  its  simplest  form.  You  throw  something  on  your  friend’s  desk  and 
ask,  “Does  this  look  right  to  you?”  You  are  asking  them  to  look  at  your  work  product  and  try  to 
find  defects  in  it.  If  you  sit  down  with  someone  and  discuss  an  approach  you  are  thinking  about 
using  in  a  design,  that  is  a  peer  review  too. 
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These  informal  types  of  peer  reviews  are  sometimes  helpful,  but  are  often  ad-hoc,  inefficient,  or 
ineffective.  A  verification  process  takes  this  concept  of  seeking  input  from  your  peers  and  formalizes  it. 
There  are  several  excellent  reasons  for  formalizing  the  verification  process: 

Assure  Peer  Reviews  Take  Place  —  Just  because  the  potential  exists  for  people  to  peer  review  each 
other’s  work  doesn’t  mean  that  they  actually  do  it.  A  formalized  verification  process  defines  what 
work  products  should  be  peer  reviewed  —  and  equally  importantly  —  when  they  should  be  reviewed. 

Select  the  Right  Reviewers  —  A  co-worker  in  the  next  office  may  be  willing  to  look  at  your  work 
product  and  give  his  opinion,  but  he  may  not  be  the  right  person  for  the  job  if  what  you  are  working 
on  is  outside  his  area  of  expertise.  This  means  that  not  only  will  you  be  wasting  his  time,  he  will  be 
wasting  yours.  And  even  worse,  you  might  actually  think  that  you’ve  got  a  solid  product  because  he 
didn’t  see  any  defects  in  it.  He  didn’t  see  them  because  he  was  not  the  right  person  to  be  looking 
for  them.  A  properly  executed  peer  review  process  selects  the  right  people  to  examine  the  product 
being  reviewed. 

Record  Defects  —  Peer  reviews  produce  a  defect  log  that  documents  all  of  the  problems  in  the 
work  product  that  need  to  be  corrected.  Walking  into  a  neighbor’s  office  and  spending  a  couple  of 
hours  discussing  a  design  isn’t  helpful  if  by  the  time  you  get  back  to  your  office  you’ve  forgotten  the 
details  of  his  observations. 

Verify  Defects  are  Corrected  —  A  list  of  defects  is  not  only  good  for  the  Producer  to  use  to 
address  the  defects  in  their  work  product;  it  is  also  good  for  someone  to  independently  verify  that 
the  defects  were  fixed.  Finding  defects  but  not  fixing  them  is  a  waste  of  resources  and  undermines 
the  whole  purpose  of  peer  reviewing. 


RULES  FOR  PEER  REVIEWS 


It  is  vitally  important  to  have  some  ground  rules  for  peer  reviews.  Criticizing  people’s  work  is  a 
touchy  subject  at  best.  Doing  it  in  a  public,  documented  forum  can  be  inflammatory  and 
frightening.  This  is  a  hurdle  that  must  be  overcome  when  introducing  a  peer  review  process  to  an 
organization.  Setting  and  enforcing  rules  is  essential.  Active  enforcement  of  these  rules  may  be 
necessary  at  first,  but  once  your  team  has  become  used  to  the  process  of  peer  reviews  this  becomes 
less  of  an  issue.  Don’t  forget  that  new  employees  will  be  in  the  same  boat  the  entire  organization 
was  when  the  process  was  first  introduced.  They  will  need  special  attention  to  get  them  introduced 
to  the  peer  review  process  that  has  become  second-nature  to  everyone  else. 

Rule  1:  No  Managers  are  Allowed  in  Peer  Reviews 

The  purpose  of  peer  reviews  is  to  find  defects,  not  to  rate  someone  for  evaluation  purposes.  Having 
a  manager  present  at  a  peer  review  may  stifle  good  communication.  Reviewers  may  be  inclined  to 
hold  back  their  criticism  for  fear  of  hurting  the  Producer’s  standing  with  their  boss.  And  it  is  tough 
enough  for  a  person  to  have  problems  in  their  work  pointed  out  to  them  by  their  peers  without  the 
person  determining  their  next  raise  sitting  in  the  room  with  them. 
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As  with  any  rule,  there  are  always  exceptions.  If  the  manager  has  special  expertise  that  is  truly 
relevant  to  the  work  product  being  examined,  an  exception  can  be  made.  This  special  need  for  their 
expertise  should  be  explained  to  the  person  whose  work  is  being  reviewed.  The  manager,  if  not 
already  familiar  with  the  peer  review  process,  should  understand  that  all  peer  reviews  find  defects, 
and  this  should  not  reflect  poorly  on  his  perception  of  the  employee’s  work. 

Rule  2:  Peer  Review  Results  are  Confidential 

Obviously  if  defects  are  recorded  and  shared  among  the  peer  review  team,  they  are  public  to  some 
extent.  But  specific  peer  review  data  should  not  be  made  generally  available  to  people  outside  the 
project  team.  Metrics  being  collected  for  process  evaluation  should  be  aggregated  when  published 
to  provide  anonymity.  The  peer  review  team  should  use  appropriate  discretion  and  keep  discussions 
confidential. 

Rule  3:  Review  Products,  Not  People 

Peer  reviews  are  intended  to  find  problems  with  work  products,  not  the  people  who  produced  them. 
Under  no  circumstances  should  personal  criticisms  be  allowed.  The  common  enemy  of  everyone  is 
the  bugs  lurking  in  the  work  products,  and  it  takes  a  team  effort  to  find  them.  This  should  be  made 
clear  to  everyone  who  participates  in  a  peer  review.  And  don’t  let  the  Reviewers  forget  that  it  will 
soon  be  their  turn  to  have  their  own  work  reviewed. 

Rule  4:  Expect  to  Find  Defects 

Once  peer  reviews  become  common,  it  will  be  obvious  to  everyone  that  all  work  products  have 
defects.  If  your  peer  reviews  aren’t  finding  defects,  you  either  have  the  wrong  Reviewers, 
insufficient  preparation,  or  you  are  targeting  the  wrong  products.  However,  when  peer  reviews  are 
new  to  the  organization  (or  a  new  employee)  it  is  a  bit  intimidating  to  have  your  peers  compile  a  list 
of  the  defects  in  your  work.  Let  everyone  know  up  front  that  every  work  product  has  defects  and 
their  objective  is  to  find  them.  Finding  these  defects  benefits  the  product,  and  therefore  the  whole 
team. 

Rule  5:  Peer  Reviews  Convey  No  Retribution 

There  should  be  no  negative  consequences  from  participating  in  a  peer  review.  Nothing  that  is 
related  to  a  peer  review  should  appear  in  evaluations  or  personnel  files.  It  is  essential  for 
participants  to  be  confident  that  what  happens  in  a  peer  review  is  strictly  a  technical  exercise  for 
improving  the  group’s  work  products  and  not  something  that  will  come  back  to  haunt  them. 
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TYPES  OF  PEER  REVIEWS 


ELSYS  utilizes  the  four  types  of  peer  reviews  briefly  described  below.  The  vast  majority  of  our 
reviews  are  either  Structured  Walkthroughs  or  Desk  Checks.  Formal  Peer  Inspections  are  very  rare. 

Formal  Peer  Inspections 

Formal  Peer  Inspections  are  typically  only  used  in  life-critical  applications.  A  Formal  Peer  Inspection 
is  a  rigorous  analysis  technique,  and  thus  is  the  most  costly  and  time  consuming  of  the  four  review 
techniques.  Formal  Peer  Inspections  must  include  domain  experts  from  outside  the  project  team. 
During  the  inspection,  defects  are  identified  and  logged,  and  action  items  are  assigned.  The  status  of 
defects  are  tracked  and  communicated  until  the  defect  has  been  resolved.  Formal  inspections  have 
the  highest  efficiency  in  terms  of  defect  removal  of  any  form  of  peer  review. 

Structured  Walkthroughs 

Structured  Walkthroughs  are  similar  to  Formal  Peer  Inspections,  but  require  fewer  Reviewers  and 
shorter  preparation  times.  A  Moderator  guides  the  other  members  through  the  review  of  the  work 
product,  with  the  Producer(s)  walking  the  group  through  the  product.  The  participants  ask 
questions  and  make  comments  about  possible  defects,  violations  of  development  standards  and 
other  problems.  During  the  Walkthrough,  defects  are  identified  and  logged,  and  action  items  are 
assigned.  The  status  of  defects  are  tracked  and  communicated  until  they  are  resolved.  Structured 
Walkthroughs  are  less  efficient  at  removing  defects  than  Inspections,  but  are  not  as  costly  and 
require  a  smaller  number  of  Reviewers. 

Desk  Checks 

Desk  Checks  are  private  reviews  carried  out  by  a  Producer  and  an  experienced  Reviewer.  The 
Producer  provides  the  Reviewer  with  a  copy  of  the  work  product  and  associated  materials.  Desk 
Checks  are  the  least  expensive  of  the  four  review  techniques.  However,  Desk  Checks  are  the  least 
effective  review  method  and  the  effectiveness  of  the  review  depends  almost  entirely  on  the 
competence  of  a  single  Reviewer. 

Round-Robin  Reviews 

Round-Robin  reviews  are  a  process  of  desk  checking  by  multiple  peers  in  a  sequential  manner.  The 
initial  Reviewer,  who  serves  as  the  Moderator,  completes  a  Desk  Check  and  then  passes  the  folder 
to  the  next  Reviewer  who  performs  another  Desk  Check,  logging  additional  defects.  This  continues 
until  all  Reviewers  have  participated  and  the  folder  is  returned  to  the  Moderator,  who  then  works 
with  the  Producer  to  ensure  that  defects  are  corrected.  Round-Robin  reviews  are  more  efficient 
than  simple  desk  checking,  and  typically  cost  less  than  Inspections  and  Walkthroughs.  However, 
Inspections  and  Walkthroughs  identify  a  greater  percentage  of  defects. 


7 


Verification  In  CMMI  Using  Peer  Reviews 

AUTHORS:  JEANNE  BALSAM,  JEAN  SWANK  LEE  SHEINER,  AND  MARK  PELLEGRINI 


PREPARING  FOR  A  PEER  REVIEW 


Having  a  successful  peer  review  is  dependent  upon  proper  preparation.  The  correct  Reviewers  must 
be  chosen,  the  review  must  be  scheduled,  and  the  work  products  to  be  reviewed  and  defect  logs 
must  be  properly  controlled. 

Choosing  Participants 

A  Reviewer  must  have  adequate  knowledge  of  the  work  product  being  reviewed.  Simply  choosing 
someone  to  be  a  Reviewer  because  they  have  the  time  to  do  it  despite  the  fact  that  they  don’t 
understand  what  they  are  reviewing  is  a  waste  of  time  and  money.  In  the  case  of  software  products, 
the  Reviewer  should  be  familiar  with  the  programming  language,  the  application  that  is  being 
developed,  and  the  system  to  which  it  is  being  integrated.  Likewise  for  hardware,  the  Reviewer 
should  understand  the  tools  and  technologies  being  used. 

It  is  also  important  for  Reviewers  to  be  able  and  willing  to  spend  adequate  time  preparing  for  the 
review.  When  they  are  invited  to  participate  in  the  peer  review  they  should  be  told  the  minimum 
amount  of  time  they  are  expected  to  spend  to  prepare  for  the  review.  If  they  cannot  commit  to  this 
amount  of  time  you  should  consider  using  another  person  as  a  Reviewer,  or  if  the  desired  Reviewer 
is  considered  essential,  rescheduling  the  review.  Although  in  some  cases  peer  reviews  can  be  done 
“cold”  (i.e.  no  advanced  preparation),  in  most  cases  reviewing  the  material  in  advance  is  essential. 
Having  a  peer  review  with  an  unprepared  Reviewer  is  generally  “just  checking  the  box,”  which  may 
give  the  appearance  of  conducting  a  peer  review,  but  is  largely  a  waste  of  time.  It  can  also  hurt  your 
peer  review  process  in  the  long  run  by  leaving  defects  undetected,  giving  the  impression  that  the 
process  is  not  worthwhile. 

A  Desk  Check  will  have  a  single  Producer  and  a  single  Reviewer.  The  choice  of  Producer(s)  is 
straightforward  —  the  person  or  people  who  created  the  work  product.  The  choice  of  Reviewer, 
however,  is  more  difficult  and  extremely  critical  if  the  peer  review  is  to  add  value  rather  than  waste 
money. 

Round-Robins,  Structured  Walkthroughs,  and  Formal  Peer  Inspections  employ  multiple  Reviewers. 
In  these  types  of  reviews  it  is  not  essential  for  each  Reviewer  to  be  an  expert  in  everything,  but 
between  them  they  should  possess  the  knowledge  to  understand  the  work  product  in  its  entirety.  It 
is  highly  desirable  to  select  at  least  one  Reviewer  from  outside  the  project  team.  Having  a  Reviewer 
with  a  totally  different  perspective  increases  the  chance  of  finding  defects  or  better  implementation 
methods.  This  also  fosters  cross-communication  between  projects. 

A  Moderator’s  primary  responsibility  is  to  run  the  peer  review  meeting.  It  is  his  job  to  be  sure  that 
all  of  the  Reviewers  participate  and  that  no  single  Reviewer  dominates  the  meeting.  The  Moderator 
must  keep  the  meeting  progressing  and  stop  off-topic  conversations  that  sometimes  are  a  natural 
consequence  of  the  discussion  of  the  work  product,  but  are  not  related  to  the  goal  of  finding 
defects.  It  is  understandable  that  people  try  to  fix  the  defects  they  find  during  the  meeting,  but  this 
is  not  its  purpose.  The  Moderator  must  prevent  problem-solving  from  displacing  the  goal  of 
problem- finding.  He  must  maintain  a  civil  atmosphere  during  the  meeting.  The  Moderator  must  be 
sure  that  the  Recorder  is  capturing  the  defects  correctly,  and  with  enough  detail  that  the  Producer 
will  understand  the  defect  later,  when  it  is  time  to  correct  the  defect. 
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Scheduling 

Peer  reviews  need  to  be  scheduled  far  enough  in  advance  to  allow  people  to  plan  around  them.  This 
means  that  the  reviews  are  normally  scheduled  before  the  work  product  is  ready  for  review, 
anticipating  the  date  that  it  will  be  ready.  If  the  work  product  is  not  ready  on  schedule,  the  amount 
that  it  slips  will  cut  into  the  cushion  that  you  have  allowed  the  Reviewers  in  their  schedule.  If  this 
happens,  you  must  either  reschedule  the  review  or  verify  that  the  Reviewers  can  adjust  their 
schedules  and  complete  the  minimum  amount  of  preparation  by  the  meeting  date.  If  they  can’t 
prepare  adequately,  a  decision  must  be  made  as  to  whether  the  investment  in  the  meeting  time  will 
still  be  a  good  one  if  the  Reviewers  go  in  with  little  or  no  advance  preparation.  Conversely,  it  is  also 
useful  to  put  a  maximum  limit  on  preparation  time,  particularly  if  budgets  are  tight.  Although  our 
experience  is  that  getting  too  little  preparation  is  more  of  a  problem  than  too  much,  you  may 
occasionally  get  someone  who  makes  preparing  for  a  review  a  full-time  job  when  you  intended  for 
them  only  to  spend  a  few  hours  doing  so. 

Reviewer  Instructions 

Reviewers  should  be  given  their  priorities  in  their  review  preparation,  particularly  if  the  time 
available  for  preparation  is  limited.  Different  Reviewers  may  have  different  fortes.  One  Reviewer 
may  be  particularly  good  at  spotting  coding  errors,  whereas  another  may  better  understand  the 
requirements  allocated  to  the  module  in  the  context  of  the  system  of  which  it  is  a  part.  For  example, 
a  piece  of  code  may  work  correctly,  but  not  do  the  job  it  needs  to  do.  This  is  a  requirements  issue. 
On  the  other  hand,  the  code  may  appear  to  be  doing  the  right  thing,  but  the  code  contains  a 
statement  that  uses  a  logical  OR  operation  when  a  bitwise  OR  was  intended.  This  is  an 
implementation  issue. 

Meeting  Length 

The  actual  peer  review  meeting  should  be  limited  to  two  hours  or  less.  After  this  length  of  time 
minds  begin  to  wander  as  fatigue  sets  in,  reducing  the  quality  of  the  review.  If  the  review  cannot  be 
completed  within  two  hours,  another  meeting  should  be  scheduled  to  complete  it.  Care  should  be 
taken  when  planning  the  meeting  so  as  not  to  attempt  to  review  too  much  material.  If  it  is  obvious 
that  the  work  product  is  too  large  to  be  reviewed  in  a  single  meeting  it  should  be  broken  down  into 
smaller  units,  if  possible,  or  scheduled  with  multiple  meetings. 

Controlling  Review  and  Reference  Material 

It  is  vitally  important  that  the  work  product  being  peer  reviewed  is  under  configuration 
management.  If  the  work  products  are  not  being  controlled  you  are  peer  reviewing  a  moving  target. 
When  Reviewers  are  told  to  begin  their  preparation  they  should  be  given  specific  revisions  of  the 
work  product  to  review.  The  forms  they  use  to  record  their  findings  should  also  document  which 
revisions  of  the  products  they  are  reviewing.  These  forms  themselves  should  be  placed  under 
configuration  management  control.  The  Reviewers  should  also  be  given  a  list  of  any  reference 
materials  they  will  need  to  adequately  assess  the  product  being  reviewed. 

Facilities 

The  facilities  in  which  the  review  takes  place  can  improve  the  convenience  and  possibly  the  quality 
of  the  review.  A  meeting  room  with  a  networked  computer  that  hosts  the  configuration 
management  tool  provides  convenient  on-line  access  to  materials  to  be  reviewed,  peer  review  forms, 
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and  reference  materials.  A  projection  system  can  be  used  to  display  the  computer  screen’s  contents 
to  all  attendees.  That  content  may  be  the  work  product  under  review,  reference  documents,  or 
defect  logs.  Easy  access  to  a  printer  capable  of  printing  multiple  size  documents  facilitates  viewing 
anything  that  is  hard  to  view  via  the  projection  system. 


CONDUCTING  A  PEER  REVIEW 


In  the  case  of  Structured  Walkthroughs  it  is  important  for  the  Moderator  to  verify  in  advance  of  the 
meeting  that  all  of  the  Reviewers  are  adequately  prepared  and  have  completed  their  defect  logs.  If 
some  Reviewers  are  not  prepared,  a  decision  must  be  made  to  hold  the  review  and  take  the  time  to 
walkthrough  the  work  product  line-by-line  (referencing  associated  documentation)  or  to  reschedule 
the  review.  This  situation  can  usually  be  avoided  if  the  Moderator  periodically  checks  with  the 
Reviewers  and  monitors  their  progress.  Sometimes  preparing  for  a  peer  review  falls  to  the  bottom 
of  the  Reviewers’  priority  lists;  being  reminded  about  it  will  usually  get  people  motivated  to  prepare. 

The  Recorder  should  take  all  of  the  individual  findings  and  combine  them  into  a  single  list  on  the 
summary  form  prior  to  the  meeting.  When  doing  this  it  is  helpful  to  keep  the  Reviewers’  initials 
associated  with  their  findings  for  easy  author  identification.  The  findings  should  be  sorted  so  that 
similar  ones  from  different  Reviewers  can  be  discussed  together.  Referencing  the  findings  by  the 
page,  line,  or  paragraph  number  of  the  work  product  makes  this  sorting  easier. 

Although  having  the  Reviewers  record  defects  in  advance  is  important,  the  meeting  should  not 
simply  focus  on  a  review  of  the  findings.  It  is  also  important  for  the  Producer(s)  to  walk  the  group 
through  their  work  product  as  this  will  often  initiate  questions  and  discussions  that  may  uncover 
defects  that  were  not  noticed  by  the  Reviewers  during  their  preparation. 

It  is  extremely  important  that  all  defects  are  captured  on  the  summary  form.  When  defects  are 
recorded  it  is  essential  that  they  are  clear  and  specific.  The  Producer  must  agree  that  what  is 
recorded  is  understandable  because  it  will  be  his  job  to  fix  the  problem.  It  is  also  important  that  the 
Reviewers  agree  that  the  defect  is  captured  accurately;  otherwise  the  Producer  will  fix  a  problem  that 
doesn’t  exist  and  fail  to  fix  the  one  that  does.  The  Moderator  needs  to  make  sure  that  the  discussion 
during  the  meeting  doesn’t  get  ahead  of  the  Recorder.  If  it  appears  that  the  group  is  generating 
findings  faster  than  the  Recorder  can  capture  them,  the  Moderator  needs  to  stop  the  conversation  to 
allow  the  Recorder  to  catch  up. 

The  purpose  of  peer  reviews  is  to  identify  problems,  not  fix  them.  Assuming  that  the  Producer  is 
competent,  he  can  fix  his  own  problems;  he  just  needs  help  in  finding  them.  Any  time  spent  in  the 
meeting  fixing  a  problem  is  time  spent  by  all  the  participants,  so  it  is  normally  not  cost-effective  to 
determine  solutions  during  the  meeting.  It  also  consumes  valuable  meeting  time,  since  the  meeting 
is  limited  to  two  hours  maximum.  Any  time  spent  fixing  a  problem  is  time  not  spent  looking  for 
other  ones.  However,  there  can  be  exceptions  to  this  rule.  If  the  work  product  is  small  enough  that 
there  is  no  question  that  there  will  be  sufficient  time  to  finish  the  peer  review,  some  latitude  can  be 
given  to  problem  solving.  Presumably  the  people  gathered  in  the  room  are  experts  in  the  product 
being  reviewed.  If  the  Producer  needs  some  assistance  in  solving  the  problem,  and  it  doesn’t  require 
too  much  time  to  discuss,  it  may  be  better  to  do  it  during  the  meeting  rather  than  try  to  reconvene 
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the  subject-matter  experts  at  another  time.  It  may  also  be  possible  that  the  problem  is  such  that  the 
Reviewers  have  a  vested  interest  in  how  it  is  solved.  In  this  case  you  are  going  to  need  their 
agreement  anyway,  so  figuring  out  how  to  solve  the  problem  during  the  meeting  may  be  appropriate. 
The  important  thing  is  to  be  sure  that  there  is  sufficient  time  to  finish  reviewing  the  material  that  is 
the  subject  of  the  review.  In  our  laboratory  we  sometimes  table  discussions  of  problem-solving  until 
the  peer  review  is  finished,  and  then  spend  the  remaining  meeting  time  discussing  solutions  for  the 
problems  for  which  it  is  appropriate  to  do  so. 

At  the  conclusion  of  the  review  a  decision  should  be  made  as  to  whether  another  review  is 
necessary.  If  the  defects  are  simple  and  straightforward  to  correct,  another  meeting  is  not  normally 
needed.  But  if  there  are  numerous  changes  needed  throughout  the  product,  or  if  large  sections  of 
entirely  new  material  need  to  be  generated  to  address  defects,  another  meeting  is  warranted. 
Sometimes  a  Structured  Walkthrough  can  be  followed  up  by  a  Desk  Check  if  the  changes  needed  are 
significant  but  not  too  overwhelming.  If  the  work  product  is  critical,  or  the  changes  are  extensive, 
another  Structured  Walkthrough  may  be  needed. 


CLOSING  A  PEER  REVIEW 


Although  preparing  for  a  peer  review  meeting  and  conducting  the  meeting  itself  is  important,  it  is  all 
for  nothing  if  the  defects  that  are  found  are  never  actually  fixed  in  the  work  product. 

Someone  must  be  designated  to  verify  that  the  work  product  is,  in  fact,  corrected  according  to  the 
findings  generated  during  the  review.  In  our  process,  the  lead  engineer  verifies  that  the  defects  have 
been  corrected  and  the  project  director  signs  the  peer  review  form  to  indicate  that  he  is  aware  of  the 
completion  of  the  peer  review.  We  also  have  a  project-independent  quality  engineer  who  verifies 
that  the  peer  reviews  have  been  closed  in  a  timely  manner. 

Our  process  requires  that  defects  must  be  closed  within  thirty  days  of  the  peer  review  meeting.  If 
there  are  defects  that  cannot  be  addressed  in  that  time-frame  and  they  can  be  deferred,  an 
engineering  change  request  (ECR)  is  created  to  document  that  the  defects  exist  and  must  be 
corrected  when  appropriate  to  do  so.  Opening  the  ECR  allows  the  peer  review  to  be  closed,  and  the 
remaining  defects  are  scheduled  for  correction  through  our  normal  engineering  change  request 
process. 
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SECONDARY  BENEFITS  OF  PEER  REVIEWS 


In  addition  to  their  primary  purpose  of  finding  defects  in  work  products,  peer  reviews  can  have 
some  very  beneficial  secondary  benefits.  Many  of  these  benefits  are  dependent  upon  a  well  designed 
and  managed  peer  review  process. 

Leverage  Team  Member  Skills 

Hopefully  you  have  at  least  one  person  in  your  organization  who  is  so  good  at  what  he  does  that 
everybody  wants  him  on  their  project;  a  “Super  Star.”  The  reality  is  the  Super  Star  cannot  work  full 
time  on  every  project  in  the  organization,  but  can  be  time-shared  to  bring  expertise  to  multiple 
projects.  The  Super  Star  will  typically  need  a  relatively  small  amount  of  time  to  review  a  design  and 
find  its  flaws  as  compared  to  the  amount  of  time  it  took  the  Producer  to  generate  the  design  in  the 
first  place.  After  it  is  fixed  it  may  be  nearly  as  good  as  if  the  Super  Star  had  done  the  whole  job 
himself. 

Share  Information  Across  Team  Boundaries 

Peer  reviews  are  a  good  opportunity  for  people  to  share  information,  particularly  when  some  of  the 
Reviewers  are  chosen  from  outside  the  project  team.  Peer  reviews  immerse  developers  who  might 
otherwise  never  look  at  a  colleague’s  work  in  a  process  where  they  must  do  so.  The  inclusion  of 
developers  from  other  project  teams  in  peer  reviews  helps  to  expose  both  teams  to  each  other’s 
work  and  expertise. 

Jump-Start  Team  Communication 

People  on  a  development  team,  particularly  if  they  don’t  really  know  one  another,  sometimes 
communicate  poorly.  A  peer  review  process  imposes  a  structure  on  them  where  they  must  not  only 
communicate,  but  do  so  in  a  manner  that  is  centered  on  constructive  criticism.  Once  they  are  used 
to  the  idea  of  critiquing  each  other’s  work,  they  will  find  it  easier  to  do  so  in  informal  situations. 

This  is  especially  helpful  to  junior  personnel,  who  might  be  uncomfortable  with  the  idea  of 
criticizing  a  more  senior  person’s  work. 

Create  Mini-Milestones 

Scheduling  a  work  product  for  a  peer  review  adds  another  milestone  to  the  product  in  addition  to  its 
completion  milestone.  The  peer  review  date  is  typically  scheduled  in  advance  of  the  completion  of 
the  work  product.  This  mini-milestone  can  motivate  the  producer  to  form  a  cohesive  work  product 
earlier  in  the  product  life-cycle,  because  it  can  be  difficult  or  embarrassing  to  reschedule  the  meeting. 

Increase  Product  Quality  from  the  Start 

If  the  producer  knows  from  the  beginning  that  his  work  will  be  examined  in  a  peer  review,  it  is  more 
likely  he  will  follow  good  practices  while  crafting  the  product.  This  benefit  extends  to  units  that  are 
not  initially  scheduled  for  peer  review  if  there  is  some  expectation  that  they  might  be  inspected  due 
to  a  planning  change. 
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Expose  Junior  Staff  to  Expert  Tutelage 

Peer  reviews  provide  an  ideal  setting  for  junior  staff  members  to  interact  with  experts  in  their  field. 
That  interaction  generally  extends  beyond  the  peer  review  through  continued  information  exchanges 
to  ensure  corrections  are  made  appropriately. 

Create  a  Team  Esprit  de  Corps 

Being  on  both  the  Producer  and  Reviewer  end  of  peer  reviews  breaks  down  barriers  and  gives  the 
team  shared  experiences.  In  addition,  if  properly  trained  and  encouraged,  the  project  team  members 
each  realize  that  finding  defects,  regardless  if  they  are  their  own  or  someone  else’s,  benefits  the 
product  and  therefore  the  whole  team. 


PRACTICAL  CONSIDERATIONS 


If  your  organization  has  not  used  a  peer  review  process  and  you  wish  to  do  so,  you  will  most  likely 
be  facing  an  initial  wall  of  resistance  that  you  will  need  to  break  through  in  order  to  be  successful.  If 
your  developers  have  not  had  other  people  closely  reviewing  their  work,  particularly  their  code,  they 
may  find  it  an  uncomfortable  idea.  They  may  not  see  the  value  in  it,  and  will  find  reasons  as  to  why 
they  do  not  have  the  time  to  prepare  for  reviewing  other  people’s  work. 

It  is  important  to  set  the  tone  correctly  right  from  the  start.  People  need  to  understand  that 
improving  work  products  is  the  point,  not  trying  to  see  who  has  the  most  (or  least)  defects  in  their 
work.  Actually,  sometimes  a  low  number  of  defects  found  is  more  of  a  reflection  on  the  quality  of 
the  Reviewers’  efforts  than  the  quality  of  the  product  reviewed.  A  cordial  and  cooperative 
atmosphere  needs  to  be  encouraged,  and  personal  criticism  cannot  be  allowed  or  it  will  poison  the 
whole  process.  The  first  reviews  may  be  a  bit  clumsy,  and  the  number  and  type  of  defects  found 
may  be  less  than  what  you  expect.  But  with  experience  things  will  get  better.  Make  sure  that  check 
lists  are  developed  and  updated  with  the  kinds  of  problems  to  look  for.  Once  people  sit  on  both 
sides  of  the  reviewer/ reviewed  fence  they  will  have  empathy  for  each  other.  And  when  people  see 
serious  problems  wrung  out  of  their  products  long  before  they  are  built  into  them,  they  will  see  the 
value  in  doing  peer  reviews.  In  ELSYS  we  went  from  having  to  nearly  twist  some  people’s  arms  to 
get  them  to  participate  to  a  point  where  peer  reviews  are  now  part  of  our  corporate  culture.  Now 
we  often  hear  people  ask,  “When  are  we  going  to  peer  review  that?” 

If  you  have  developers  who  are  completely  unwilling  to  participate  in  peer  reviews  with  the  proper 
attitude  it  is  probably  better  to  exclude  them  from  the  process  at  first.  Get  people  with  positive 
attitudes  to  pioneer  your  process.  You  will  undoubtedly  have  some  bugs  in  your  processes  that 
you’ll  need  to  iron  out,  so  it  is  far  better  to  include  people  who  want  to  make  the  process  work. 

Once  you  have  a  cadre  of  people  who  are  reaping  the  benefits  the  word  will  get  out.  The  naysayers 
will  see  their  colleagues  building  better  products  with  fewer  problems,  and  that  will  encourage  them 
to  participate  in  the  peer  review  process  as  well. 
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SUMMARY 


A  peer  reviews  is  a  tool,  but  like  any  tool,  the  idea  is  to  get  more  work  out  of  it  than  you  put  into  it. 
And  if  you  don’t  know  how  to  use  the  tool,  or  choose  to  use  it  incorrectly,  your  effort  will  be 
wasted. 

Here  are  some  things  you  should  keep  in  mind  when  designing,  implementing,  and  practicing  a  peer 
review  process: 

•  The  most  important  goal  of  a  peer  review  is  to  find  defects 

•  Keep  managers  out  of  peer  reviews  unless  they  have  technical  knowledge  that  is  essential 

•  Keep  peer  review  results  confidential  within  the  team  to  the  greatest  extent  practical 

•  Remember  that  you  are  reviewing  products,  not  the  people  who  produced  them 

•  Expect  to  find  defects  —  nobody  is  perfect. . .  nobody! 

•  Do  not  use  peer  review  results  to  punish  people 

•  Choose  participants  who  have  the  expertise  to  be  effective  Reviewers 

•  Get  a  commitment  from  Reviewers  to  spend  an  appropriate  amount  of  time  preparing 

•  Schedule  meetings  to  allow  sufficient  time  for  Reviewers  to  prepare 

•  Place  any  work  product  that  is  being  reviewed  under  configuration  control 

•  Keep  peer  review  meetings  under  two  hours  in  length.  Schedule  another  session  if 
needed 

•  Keep  the  meeting  moving,  and  on  topic 

•  For  Structured  Walkthroughs,  provide  a  meeting  room  with  facilities  that  support  the 
review 

•  Provide  Reviewers  with  any  needed  reference  material 

•  Instruct  Reviewers  where  to  concentrate  their  examination,  if  appropriate  to  do  so 

•  Identify  defects  in  a  peer  review,  don’t  fix  them,  unless  special  circumstances  warrant  it 

•  Make  sure  that  all  defects  are  recorded  clearly  and  accurately 

•  Verify  that  all  defects  from  a  peer  review  are  closed  appropriately 

•  Maintain  a  cordial  atmosphere  during  peer  review  meetings;  do  not  tolerate  personal 
criticisms 

•  Select  an  appropriate  peer  review  type  for  the  product  being  reviewed 

•  Peer  review  requirements  before  design  begins;  peer  review  designs  before 
implementation 

•  Peer  review,  at  a  minimum,  complex  implementations  and/ or  critical  components 

•  Peer  review  test  procedures 

•  Peer  review  critical-path  items 

•  Foster  an  atmosphere  where  the  team  understands  that  finding  defects  is  a  good  thing 
that  benefits  everyone. 
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How  Has  CMMI  Improved  Our  Program  and  Project  Performance  -  Or  Has  It? 


Integrated  Defense  Systems 


Topics 

•  CMMI  Deployment  and  Use 

•  Common  Process  Initiatives 

•  Plans  for  SE  Management  on  Programs 
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CMMI  Maturity  Level  -  All  IDS  Business  Units  and  All  Major  Sites 

Boeing  Sites  Appraised  at  ML5 


at  most  sites  in  SW,  SE,  IPPD, 
and  Supplier  Management. 

Boeing  Is  Committed  to  CMMI  I 
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Baldrige  Notional  Qualify  Program 


Overarching  Framework 

•  Management  Leadership 

•  High  level  visibility 


Registration 

•  Process  documentation 
standard 

•  Quality  measurement 
system 

Boeing  Proprietary 

Lbun 

Lean  Enterprise 

“How  to”  Strategies 

•  End  to  End 

•  Value  Stream  Analysis 


Business 

Excellence 


Program  Management 
Best  Practices 
Management 

•  Maturity  matrix 

•  Review  feedback 


3M 


Capability  Maturity 
Model  Integration 
Process  Improvement 

•  Implementation 

•  Metrics  driven 
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Concept  of  SE  Execution 

•  Use  CMMI  in  its  fullness  to  manage  process  maturity 

•  Incorporate  Program  SE  Performance  Management 

-Work-product  metrics  and  trends 

-  Methods  for  using  assessments  and  metrics  to 
manage  the  program 

•  Create  a  customer/contractor  partnership  to  develop 
“structured  management”  on  the  programs 


Focus  on  SE  Practice  and  Related  Measures 
to  Ensure  Program  Success 
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Assessment  Initiatives  Hierarchy 


PMBP  (Boeing) 

•  Program  Management 

•  Supplier  Integration 

•  Organization 

•  Business  Plan 

•  Program  Execution 

•  Risk  and  Issue  Mgt. 


Baldrige 

•  Customer  /  Market 
Focus 

•  Leadership 

•  HR  Focus 

•  Business  Results 


ISO  9000 

•  Customer  &  Market  Focus 

•  Manage  Business 

•  Process  and  Product  Quality 


Assessments 

•  Expands  on  CMMI 

•  Management 
Commitment 

•  Infrastructure 

•  Program  Execution 

•  Monitor  and  Support 

•  Performed  on  all  programs 

•  Done  through  complete  Life 


Cycle 


Baldrige 


ISO  9000 


PMBP 


CMMI 


Domain  Assessments 


ft 

Program  Teams 


CMMI 

•  Best  Practices  for  selected 
Process  Areas 

•  Engineering 

•  Project  Management 

•  Supplier  Management 

•  Framework  for  process 
improvement 

•  Appraisal  Process 
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Improvement  Initiatives 


Goal:  Productivity 

A 
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DFMA  %/ 
MBD/E  \ 
Simulation  ^ 


Goal:  Discipline 

CMMI/SE 

ISO  9000 /AS  91 00 


Baldrige  Quality  Awar 


Integrated  Product  Teams 
Management  Best  Practices  ^ 
HILT  Ownership  ^ 

y 

Goal:  Performance 


Common  Processes/ cf 

<c 

Common  Tools/  ^ 

f 

s? 

Goal:  Efficiency  & 
Flexibility 


Engineering 

Excellence 


Boeing  Has  Defined  Common  Processes 
Consistent  with  CMMI 
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Applying  Common  Process  Assets  on  Programs 


O) 

CD  "§ 
<D 

.E  o 

O)  £= 

c  c 

LLJ  O 

C n  ° 
□ 


Process  Breakdown 
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Process  assets  (methods, 
process  instructions, 
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Program  Unique 
Processes 
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Cross-functional 
process  threads 
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Value  Stream,  Business  Scenario,  Process  Thread 


Legend 

Common  process 

Program  unique 
used  at  multiple  sites 

Program  unique 
used  at  single  site 


Mission  Define  Concept 
Analysis  Mission  P 
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Program  Lifecycle 
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Strengthening  SE  Execution  at  IDS 


Monitor  and 
Support 


Management 

Commitment 


•  Boeing 

Strategy/Policies 

•  Leadership 

•  Resources 


Disciplined 
assessment  process 
Leading  indicators 


Implementation/ 

Execution 


Planning 
Resources 
Organization  /  RAA 


•  Comprehensive  SE 
process  assets 

•  Subject  Matter  Experts 

•  SE  integration  across 
sites 
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How  Has  CMMI  Improved  Our  Program  and  Project  Performance  -  Or  Has  It? 


Summary 

•  Maturity  level  and  Capability  profile  are  necessary, 
but  not  sufficient  for  SE  maturity 

•  Recommend  deployment  of  SE  processes  and 
metrics  to  ensure  program’s  ability  to  execute 
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Integrated  Defense  Systems 
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High  performance.  Delivered 


An  Enterprise  Wide  CMMI  Implementation 
at  Accenture  by  Sarah  Bengzon 


Learning  Objectives 


•  About  Accenture 

•  Business  Context 

•  Implementation  Scope 

•  Key  Program  Components 

•  Guiding  Principles 
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About  Accenture  (NYSE:  ACN) 


Accenture  is  a  management  consulting,  technology  services,  and 
outsourcing  company.  Committed  to  delivering  innovation, 
Accenture  collaborates  with  its  clients  to  help  them  become  high- 
performance  businesses  and  governments. 


Comms  & 

Financial 

Government 

Products 

Resources 

High  Tech 

Services 

•  Communications 

•  Banking 

Serving  sectors: 

•  Automotive 

•  Chemicals 

•  Electronics  & 

High  Tech  | 

•  Media  & 
Entertainment 

•  Capital  Markets 

•  Insurance 

•  Defense 

•  Postal 

•  Education 

•  Revenue 

•  Human  Services 

•  Immigration 

•  Justice/Security 

•  Election  Services 

•  Health  Services 

•  Industrial 

Equipment 

•  Pharmaceuticals  & 
Medical  Products 

•  Retail  &  Consumer 

•  Transportation  & 
Travel  Services 

•  Energy 

•  Forest  Products 

•  Metals  &  Mining 

•  Utilities 
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Business  Context 


In  a  changing  business  context,  the  cost  of  not  delivering  quality 
services  is  high  and  can  have  a  significantly  impact  to  the 
business. 

•  Market  growth 

•  Bigger,  more  complex  programs 

•  Offshore  components 

•  Growing,  diverse  workforce 

•  Increasing  competition 
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Implementation  Scope 


To  operate  as  a  high  performing  business,  Accenture  needs  to 
operate  with  process  rigor  and  consistency  across  a  global  and 
complex  business  model. 

•  All  Operating  Groups 

•  All  geographies 

•  Multi-lingual,  multi-cultural 

•  Complex  business  and 
technology  solutions 

•  CMMI  SW/  SE/  IPPD 
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Key  Program  Components 


•  Governance  and  Program 
Management 

•  Awareness  and  Sponsorship 
Building 

•  Mobilization 

•  Deployment 

•  Process  Improvements 

•  Measurement  and  Assessments 


Copyright  ©  2005  Accenture  All  Rights  Reserved. 


Governance  and  Program  Management 


•  Senior  Executive  Governance 

-  Provide  overall  direction  and  leadership 

-  Provide  key  decision  making 

•  Small  Central  Program  Management 
Team 

-  Define  and  maintain  policies,  process 
assets 

-  Provide  standard  training  and 
communications 

-  Coordinate  appraisals 

-  Facilitate  best  practice  sharing 

-  Support  deployment 

•  Larger  ‘Local’  Implementation  Teams 

-  Develop  local  quality  plans 

-  Tailor  standard  assets 

-  Implement  training  and  communications 
plan 

-  Implement  quality  program 

-  Share  best  practices  and  experiences 
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Acceptance 


Awareness  and  Sponsorship  Building 


► 
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Mobilization 


Planning 

QPI  team  sourcing 

Monthly  training 

QPI  communications 

Standard  QPI  processes  and 
tools 

QPI  deployment  support  and 
cross-OG  issue  resolution 

Note:  QPI  (Quality  and  Process  Improvement) 
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Deployment 


•  Provides  projects  with 
standard  processes,  tools, 
coaching,  and  training  on 
systems/  software 
engineering  and  project 
management  disciplines 

•  Provides  coaching  and 
mentoring 

•  Provides  monthly  reviews 
against  best  practices 

•  Provides  increased  visibility 
into  project  execution 


Copyright  ©  2005  Accenture  All  Rights  Reserved. 


Provide  better  value 
in  our  service  to 
clients 


Executive 
sorship 


Improve 

predictability 

of 

performance 
and  support 


Engagement 

Teams 


Leverage  the  management 
and  technical  knowledge 
from  past  engagements 


QPI  Program 


Address 

Accenture  quality 
requirements  and 
best  practices 


KEY 


Objectives 

Enablers 
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Process  Improvements 


As  we  continue  our  enterprise  wide  CMMI  implementation,  there  is  a 
continuous  improvement  cycle  to  standard  capability  infrastructure. 


METHODS 

MEASUREMENTS 

Copyright  ©  2005  Accenture  All  Rights  Reserved. 
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Process  Improvements  -  Methods 


All  Accenture  people,  regardless  of  where  they  are  located,  use 
the  same  methodology.  This  gives  us  the  ability  to  move  work  to 
the  most  capable  and  cost  effective  location(s). 

•  Common  language 

•  Distributed  work  model 

•  Standard  transition  points 

•  Guidelines  for  planning  and 
managing  distributed  work 


Accenture  Delivery  Methods 
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Process  Improvements  -  Training 


Best  practices  are  woven  into  standard  methods  and  deploying 
via  enterprise  wide  training  curriculum. 


View  course 
catalog  by 
category,  title, 
or  certificate 


Curriculum 
allows  each 
employee  to 
see  courses 
specifically 
required  or 
recommended 
for  them 


Contact  Us  Preferences  Help  Search:  “  ^  ***'**  “ 

|  Go  Advanced  Search 

■■D 

> 

myLea 

C€ nture  View  CO u rse  cata log' 

View  curriculum  View  transcript’-  Plan  my  developmenv 

Roles’- 

Transcript 
shows  courses 
taken,  waived, 
denied,  or 
cancelled. 


E  Government 
EjD-  Products 
E  Resources 

■■  2}  Consultint^W^fTforce  -  Service  Line 
3^pi^nse  Workforce  (Business  Practices) 

)  Services  Workforce 
■  5)  Professional  Skills 
+  Baseline  Shareholder  Value 
+  Desktop  Applications 
3  Functional  Business  Acumen 
+  Global  Quality  Management 
-  Leadership  &  Personal  Development 

[j  Collaboration/Teaming/Interpersonal  Dynamics 


Developing  Other 


E-  Leadership  (L&PD) 

+  Personal  Productivity  &  Effectiveness 
ED-  Structured  Communication 
+  Program  and  Project  Management 
+]  Sales 

6)  Geographies 

7)  Virtual  Seminars 

8)  SkillSoft 


Find  a  course  by  category 


Create  Link 


] 


Title 

Coaching  Handbook 
Coaching  in  the  Workplace 
Creative  Edge  (Classroom) 

Essential  Skills  for  Tomorrow's  Managers:  A  Manager's  Primer. , .  Online 
Leaders  Window  Classroom 

Leadership  Skills  For  New  Supervisors  Classroom 


Development 
plans  shows 
career 

aspirations,  skill 
needs,  and 
actions  taken 
and  planned. 
Career  advisors 
review  and 
indicate 
approval  for  all 
actions. 
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Measurement  and  Assessments 


•  Multiple  baselines 
appraisals  in  6  months 
covering  all  operating 
groups  and  geographies 

•  Multiple  CMMI  service 
providers 

•  Reuse  of  separate 
organizational  appraisal  for 
common  infrastructure 
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Guiding  principles 


•  Build  once,  implement  consistently. 

•  Balance  consistency  with  flexibility. 

•  Use  and  reuse  what  is  available. 

•  Common  approach  for  select  geographies 
and  types  of  work. 

•  Integration  with  other  corporate  programs. 

•  Strong  governance  model. 

•  Consistent  appraisals  and  actions. 
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High  performance.  Delivered. 


Questions? 


Sarah  Bengzon,  Director  of  Quality 

Tel:  +1.703.947.1030 

Email :  sarah. bengzon@accenture.com 


I  Shaping  your  Processes 

LU  k«/0  for  Competitive  Advantage 


Understanding  “Why?” 


David  N.  Card 
dca@q-labs.  com 

Based  on  D.  Card,  Understanding  Causal  Systems,  Crosstalk ,  October  2004 
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Shaping  your  Processes  for  Competitive  Advantage 


Agenda 


•  The  Problem 

•  Basic  Concepts 

•  Defect  Classifications 

•  Causal  Analysis  in  the  CMM  and 
CMMI® 

•  An  Opportunity 

•  Summary 

CMM  and  CMMI®  are  registered  trademarks  of  Carnegie  Mellon  University 
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Shaping  yotir  Processes  for  Competitive  Advantage 


The  Problem 

•  Many  of  the  potential  benefits  of 
measurement  and  analysis  activities 
depend  on  effective  causal  analysis 

•  Causal  analysis  often  produces 
superficial  or  no  meaningful  action,  even 
in  “mature”  organizations 
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Shaping  your  Processes  for  Competitive  Advantage 


Examples  of  Weak  Results 

•  Identified  cause  does  not  lead  to  any  action 

-  Bad  data 

-  Personnel  issues 

•  Causes  and  actions  are  superficial 

-  Defect  rates  from  inspections  are  low,  so  reinspect 

-  Defect  rates  from  inspections  are  high,  so  orient  the 
producer 

•  Only  a  small  number  of  problems  may  result  in 
false  OOC  signals  or  OBE  (overcome  by  events) 
situations 

•  Tendency  to  stop  at  “first  plausible  explanation” 
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Shaping  yotir  Processes  for  Competitive  Advantage 


Contributors  to  the  Problem 

•  Misunderstanding  of  basic  concepts 

-  Causality 

-  Causal  system 

•  Inadequate  defect  classification 
schemes 


•  Ad  hoc  causal  analysis  processes 

-  Bad  habits 

-  Differences  between  CMM  and  CMMI 
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Shaping  your  Processes  for  Competitive  Advantage 


Causal  Analysis 


•  Examination  of  information  about  problems,  with 

•  Intent  to  identify  causes  of  defects  so  that  they  can 
be  prevented  or  detected  earlier,  or  so  that 
appropriate  corrective  action  can  be  taken 

•  Many  different  approaches,  called  defect  causal 
analysis  or  root  cause  analysis,  employ  many 
different  techniques 

•  Performed  in  response  to  an  anomaly  or  as  part  of 
a  continual  improvement  program 
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Shaping  yoiir  Processes  for  Competitive  Advantage 


Concept  of  Causality 

•  Conditions  of  causality 

-  Cause  and  effect  must  demonstrate 
association  or  correlation 

-  Cause  must  precede  the  effect  in  time 

-  Mechanism  by  which  the  cause  produces 
the  effect  must  be  identified 

•  Assignment  of  cause  in  a  “human¬ 
intensive  system”  always  includes 
significant  element  of  subjectivity 
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Shaping  your  Processes  for  Competitive  Advantage 


A  Causal  Relationship? 
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Shaping  your  Processes  for  Competitive  Advantage 


Causal  System 

•  Network  of  interacting  factors  that  affect  an 
outcome  of  interest 

-  Causes  may  linked  hierarchically  or 
laterally  —  causes  may  be  effects,  too! 


-  A  vocabulary  limited  to  cause  and  effect 
usually  isn’t  sufficient  for  reasoning  about  a 
causal  system 
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Shaping  your  Processes  for  Competitive  Advantage 


Terminology  for  Causal  Analysis 


•  The  problem  is  the 
critical  issue 

•  Symptoms  usually  are 
the  most  readily  visible 
consequences  of  the 
problem 

•  Causes  contribute  to  the 
occurrence  of  the 
problem 

•  Systematic  problems 
are  those  that  repeat 
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Shaping  your  Processes  for  Competitive  Advantage 


Elements  of  a  Causal  System 


Observations 


Cause 


Problem 


Preventive 


Corrective 


Symptom 


Mitigating 


Actions 


•  Action  may  be  taken  on  many  different  elements  of  a 
causal  system 

•  Selecting  the  right  action  depends  on  the  cost  of  the 
action  and  the  expected  impact  on  the  system 
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Shaping  your  Processes  for  Competitive  Advantage 


Defect  Classifications 


•  Meaningful  classifications  are  essential  to  identify 
trends  and  “systematic  errors” 

•  Most  common  dimensions  include: 

-  when  inserted  (activity) 

-  when  found  (activity) 

-  type  of  error  made 

•  Type  of  error  may  be  specific  to  the  work  product 
or  phase 

•  Classifications  are  intended  as  a  tool  for  gaining 
insight 

-  May  require  customization  to  problem  domain 

-  Must  be  understandable  to  engineers 
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Shaping  ycur  Processes  for  Competitive  Advantage 

“Ideal”  Attributes  of 
Classifications 

•  Orthogonal  Dimensions 

•  Mutually  Exclusive  Categories  within  a 
Dimensions 

•  Objective  Criteria  for  Assigning 
Categories 

•  Sensitivity  to  Behavior  -  changes  in 
behavior  result  in  changes  in  meaures 
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Shaping  your  Processes  for  Competitive  Advantage 


Example  Pareto  Chart 


INTERFACE 

| - 1  DATA 

i - 1  LOGIC 


INITIALIZATION 
I  I  COMPUTATION 


Type  of  Defect 

NASA  Software  Engineering  Laboratory  Classification 
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Causal  Analysis  Process 

•  Causal  analysis  assumed  to  be  “intuitive” 

•  Processes,  procedures,  and  tools  often  minimal 

•  Insufficient  emphasis  on  ensuring  that  the  right 
people  participate 

•  CMM/CMMI-required  “structure”  added  later 


F  r  -a  -n  c  e 


G  e  r  im  a  ny 


Sweden 


United  Kingdom 


United  States 
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Shaping  yoiir  Processes  for  Competitive  Advantage 


Relationship  to  CMM 

•  Level  4  —  Defect  Causal  Analysis 

-  May  be  ad  hoc,  bad  habit! 

-  Performed  in  response  to  out  of  control 
situations 

•  Level  5  —  Defect  Prevention 

-  A  Key  Process  Area  (KPA)  of  CMM 

-  Systematic  approach  required  for  DCA  -  “in 
accordance  with  a  documented  procedure” 

-  Performed  even  when  process  is  in  control 

-  Additional  planning  and  feedback  requirements 


F  r  -a  -n  c  e 
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Shaping  yotir  Processes  for  Competitive  Advantage 


DP  Planning 

•  Based  on  results  of  process  performance  analysis 
provided  by  (Quantitative  Process  Management 
(QPM),  Software  Quality  Management  (SQM), 
Process  Change  Management  (PCM)  activities 

•  Defines 

-  Focus  of  DP  activities  (e.g.,  problem  area) 

-  Charter,  composition,  roles,  and  responsibilities 
of  defect  causal  analysis  team(s) 

-  Charter,  composition,  roles,  and  responsibility  of 
action  team(s) 

-  Schedules  for  phase  kickoff  meetings 

•  May  not  address  all  project  activities  and  products 
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Phase  Kickoff  Meeting 

•  Provides  regular  feedback  from  DCA  sessions 

•  Entire  project  staff  participates 

•  Typical  topics 

-  Lessons  learned  (Dos  and  Don’ts)  from 
previous  projects  and  builds 

-  Defect  causal  analysis  and  other  process 
improvement  activities  to  be  conducted 

-  Goals  and  objectives  for  this  phase 

-  Changes  to  methods  and  tools  for  this  phase 


F  r  -a  -n  c  e 
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Causal  Analysis  and  Resolution 

•  CMMI  Process  Area  at  Level  5 

•  Differences  from  CMM  DP 

-  Phase  Kick-off  Meetings  not  addressed 

-  Planning  requirements  relaxed 
(management  versus  technical  plan) 

-  Scope  broadened  to  include  all  types  of 
anomalies,  not  just  defects 

•  DP  provides  the  more  challenging  set  of 
requirements 
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Shaping  yotir  Processes  for  Competitive  Advantage 


Relationship  to  Six  Sigma 

•  Many  causal  analysis  techniques  provided  in 
typical  Six  Sigma  training  programs  (e.g,  Error 
Modes  and  Effects  Analysis) 

•  Defect  prevention  planning  and  team-based 
approach  to  DCA  (CMM  requirements)  usually  are 
not  explicit  elements  of  Six  Sigma 

•  DP  in  the  SW-CMM,  and  CAR  in  the  CMMI, 
assume  processes  are  defined;  the  need  to  define 
processes  prior  to  DCA  increases  the  time  and 
effort  required 


F  r  -a  -n  c  e 
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Shaping  your  Processes  for  Competitive  Advantage 


Opportunity  -  IEEE  1044 

•  IEEE  Standard  1 044  -  Classification  of  Software 
Anomalies  (1995) 

•  Working  group  established  to  revise  this  standard 

•  Goals  of  revision 

-  Incorporate  current  concepts 

•  Inspection  defects 

•  Orthogonal  defect  classification 

•  Defect  causal  analysis 

•  CMMI,  Six  Sigma,  etc. 

-  Extend  to  defect  prevention  and  improvement  from  just 
problem  management 

•  Some  face-to-face  meetings,  but  most  work  to  be 
accomplished  off-line 


F  r  -a  -n  c  e 
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Summary 


•  Regular  and  effective  causal  analysis  is  an 
essential  element  of  any  continuous  improvement 
program 

•  Many  concepts  of  causal  analysis  are 
misunderstood 


DP  (CMM)  and  CAR  (CMMI)  requirements  differ  in 
some  important  ways 

Do  causal  analysis  right  -  from  the  start! 


-  Training 

-  Good  classifications 

-  Systematic  process 
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About  Q-Labs 


■  Consulting  and  Appraisals  in 
Software  Measurement, 
CMM/CMMI,  ISO  9000, 
SPICE,  etc. 

■  International  Company 


■  A  broad  international  client 
base,  e.g. 

■  Alcatel,  Bouygues  Telecom, 
France  Telecom,  Orange 

■  AXA,  BNP  Paribas,  Banques 
Populaires 


■  France 

■  Germany 

■  Sweden 

■  UK 

■  USA 

■  120  employees 

■  ISO  9001  Certified 


ABB,  R.  Bosch,  EDF,  IBM, 
Siemens,  Schneider  Electric, 
Thomson  Detexis,  Volvo, 
Sony 

Atomic  Energy  Board  of 
Canada,  FAA,  Norwegian 
Ministry  of  Justice,  Swedish 
Civil  Aviation  Administration 

Thales,  Thomson,  FMV,  US 
Army  TACOM 
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ABB  Overview 


■  Leader  in  power  and  automation 
technologies 

■  Enable  utility  and  industry 
customers  to  improve  performance 
while  lowering  environmental 
impact 

■  ABB’s  products  help  operate 
Utilities,  process  industries, 
manufacturing  plants,  and  other 
industries 


■  Present  in  over  1 20  countries 
and  employs  1 10,000  people 

■  First  company  in  the  world  to 
sell  100,000  robots 

■  A  vast  majority  of  ABB  products 
have  software  &  hardware 

components 
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ABB’s  Organizational  Structure 


Corporate  Research 


IS  Group  Services 


Power  Products 


Power  Systems 


A  III! 
*M»IP 


Automation  Products 


Process  Automation 


CO 


Robotics 


a  nil 
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Organizational  Structure  of  ABB 


Power  Technologies  Segment 

■  Power  Systems 

■  Medium-Voltage  Products 

■  High  Voltage  Products 

■  Transformers 

■  Utility  Automation  Systems 

Automation  Technologies 
Segment 

■  Automation  Products 

■  Manufacturing  Automation 

■  Process  Automation 
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ABB’s  Products 


LO 


A  »» 


■  Power  Products 

■  Power  Systems 
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ABB’s  Products 


CD 


■  Automation  Products 

■  Process  Automation 

■  Robotics 
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ASPI 


ABB 

SOFTWARE 

PROCESS 

INITIATIVE 


ABB  Software  Process  Initiative  (ASPI) 
acts  as  the  Corporate  Engineering 
Process  Group 

ASPI  is  composed  of  members  from  2 
ABB  Corporate  Research  Centers 
(CRCs): 

■  United  States:  Raleigh 


■  Sweden:  Vasteras 


Responsible  for: 

■  Initiation  activities 

■  Performance  of  appraisals 

■  Development  of  improvement 
methodologies, 

■  Evaluation  and  deployment  of  pilots  within 
ABB  for  CMMI  transition,  PSP/TSP,  etc. 

■  Assisting  units  in  establishing  improvement 
plans  and  acting 

■  Collect  lessons  learned  from  process 
improvement  activities 


Alin 
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ABB  Corporate  EPG  Support 


Support  ABB  Development  Units  in  their  Continuous  Improvement 
Efforts  to  establish  a  culture  of  product  development  excellence 


CEPG 


SECRC  ASPI  Team 
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Continuous  Process  Improvement  Cycle 

■  Initiate  Improvement  activity 

■  Define  Medium/Long-term  Strategic  Improvement  Plan  (SIP)  and 
identify  organization’s  business  goals 

■  Conduct  internal  CMMI  Appraisal  (Class  B) 

■  Develop  Process  Improvement  Plan  (PIP) 

■  Prioritize  process  improvement  activities  using  Business  Objectives 

■  Implement  PIP 

■  Monitor  ROI 

■  Re-Initiate 


Alin 
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Results  of  Internal  ABB  Class  B  CMMI  Appraisal 


■  Establishes  a  baseline  in 
the  organization 

■  Serves  as  a  basis  to 
identify  process 
improvement  activities 

■  Recommended  to  include 
the  Measurement  and 
Analysis  Process  Area 


Generic  Goal  2 


Practice 

RD 

ReqM 

PP 

PMC 

MA 

SAM 

Ver 

PPQA 

CM 

Specific  Goal  1 

SP1.1 

Medium 

Medium 

High 

High 

High 

High 

■ 

High! 

SP  1.2 

High 

Medium 

High 

High 

High 

High 

Highl 

SP  1.3 

High 

High 

High 

High| 

SP  1.4 

High 

SP  1.5 

High 

High 

SP  1.6 

High 

SP  1.7 

High 

Specific  Goal  2 

SP  2.1 

High 

Medium 

Medium 

High 

High 

High 

High 

SP  2.2 

High 

High 

High 

High 

High 

SP  2.3 

High 

High 

High 

High 

High 

SP  2.4 

High 

High 

High 

SP  2.5 

Medium 

SP  2.6 

High 

SP  2.7 

Specific  Goal  3 

SP  3.1 

Medium 

High 

High 

High 

SP  3.2 

Medium 

High 

■■ 

SP  3.3 

High 

High 

SP  3.4 

High 

SP  3.5 

Medium 
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GQM  Definitions 


■  Define  major  goals  of  the  process  improvement  activity 


■  Questions  derived  from  goals  that  must  be  answered  to 
determine  if  the  goals  are  achieved 


■  Measurements  that  provide  the  most  appropriate 
information  for  answering  the  identified  questions 
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Example  of  GQM  for  Process  Measurement 


Goals 


Questions 


Metrics 


Decrease 
Development  Costs 


How  many  defects 
products  have? 


Number  of  defects 
found  in  released 
products  in  first 
three  months,  by 
month 


Improve  Quality  of 
Products 


Ensure  on-time 
Delivery 


Decrease  Risks  in 
Projects 


Alin 
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Using  Analytical  Hierarchy  Process  (AHP)* 


■  AHP  provides  a  way  to  prioritize  alternatives  to  satisfy  an 
organization’s  business  goals 

■  It  represents  a  decision-making  problem  in  hierarchical  structure 
and  consists  of: 


■  A  business  goal 

■  Criteria  to  satisfy  the  business  goal 

■  Alternatives  to  select  to  achieve  the  business  goal 

The  GQM  approach  can  be  used  to  derive  tree  in  AHP 


*  Golden,  B.  L.,  E.A.  Wasil,  P.T.  Harker, 
“The  Analytic  Hierarchy  Process”, 

Springer-Verlag,  1989 


© 


G03l 
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AHP  Steps 


1 .  Setup  criteria  from  cause  of  defects 

2.  Setup  alternatives  from  specific  practices 

3.  Assign  weights  to  criteria 

4.  Perform  pair-wise  comparisons  of  alternatives 
for  each  criterion 

5.  Determine  the  priority  of  alternatives 


Alin 


Discussion  of  an  Example  at  ABB 


To  follow  Example  please  refer 

to  provided  handouts 
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Determine  Goal 


■  Reduce  the  Cost  of  Poor  Quality 

I 

■  Reduce  the  number  of  defects  found  at  Integration  Testing  stage 

l 

■  Reduce  rework  time 

j 

■  Reduce  cost  associated  with  rework  time  by  $150  kUSD 


Alin 
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Step  1 :  Set  Up  Criteria 


■  Couple  of  salient  known  recent  issues  to  the 
organization  include: 

■  There  have  been  issues  with  capturing  and  managing 
requirements  properly  -  this  issue  has  been  noticed  especially 
when  requirements  have  been  provided  to  supplier 

■  Quality  with  primary  software  sub-contractor  has  been  a 
problem  lately  as  well 

■  Criteria  then  are  developed 

■  Supplier  agreements  do  not  always  reflect  requirements 

■  Requirements  not  properly  defined 

■  Requirements  not  properly  managed 


Alin 


ABB  Inc.,  USCRC  -  18 


. . .  Step  1 :  Set  Up  Criteria 


Table  of  criteria 


Criteria 


Supplier  Agreements  do  not  always  reflect 
requirements _ 

Requirements  not  properly  defined _ 

Requirements  not  properly  managed 


Alin 
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Step  2:  Set  Up  Alternatives 


■  From  the  appraisal  conducted,  the 
PAs  identified  at  the  site  as  “defect 
generation  responsible”  include: 

■  RD,  REQM,  SAM,  VER,  and  CM 

■  With  the  above  issues  in  mind,  and 
the  appraisal  results  identified,  the 
PAs  that  contribute  most  to  the 
criteria  identified  are  established: 

■  SAM,  RD,  and  REQM 

■  MA  will  be  a  “must” 


Practice 

RD 

ReqM 

PP 

PMC 

MA 

SAM 

Ver 

PPQA 

CM 

Specific  Goal  1 

SP  1.1 

Medium 

Medium 

High 

High 

High 

High 

High 

SP  1.2 

High 

Medium 

High 

High 

High 

High 

High 

SP  1.3 

High 

High 

High 

High  | 

SP  1.4 

High 

SP  1.5 

High 

High 

SP  1.6 

High 

SP  1.7 

High 

Specific  Goal  2 

SP  2.1 

High 

Medium 

Medium 

High 

High 

High 

High 

SP  2.2 

High 

High 

High 

High 

High 

SP  2.3 

High 

High 

High 

High 

High 

SP  2.4 

High 

High 

High 

SP  2.5 

Medium 

SP  2.6 

High 

SP  2.7 

Specific  Goal  3 

SP  3.1 

Medium 

High 

High 

High 

SP  3.2 

Medium 

High 

SP  3.3 

High 

High 

SP  3.4 

High 

SP  3.5 

Medium 

Generic  Goal  2 
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Step  2:  Set  Up  Alternatives 


From  the  PAs  identified  as  most 
likely  contributors  to  the  criteria, 
and  referring  to  the  findings  after 
the  appraisal  was  conducted,  the 
description  of  recommendations 
associated  with  specific  practices 
of  the  PAs  identified  are  presented 
as  shown  in  this  slide 


Practices 

Description  of  Alternatives 

RDSP1.1-1,  -2 

Consistently  elicit  and  document 
customers  needs  and  expectations 

RDSP  1.2 

Consistently  develop  and  document 
MRS 

RDSP2.1 

Consistently  document  ERS  and 
establish  link  between  ERS  and  MRS 

RD  SP3.2 

Consistently  establish  and  maintain 
definition  of  required  functionality  in 

ERS 

RDSP  3.3 

Analyze  completeness  and  sufficiency 
of  requirements  ina  consistent  fashion 

RDSP  3.5 

Consistenlty  validate  product 
requirements 

RM  SP1.4 

Establish  and  maintain  bi-directional 
traceability 

RM  SP1.5 

Identify  and  document  inconsistencies 
between  work  products  and 
requirements 

SAM  SP1.2 

Select  Suppliers  based  on  their  ability 
to  satisfy  requirements 

SAM  SP  2.1 

Review  COTS  products  to  ensure  they 
satisfy  requirements 

SAM  SP2.3 

Accept  acquired  product  verifying  it 
meets  requirements 

Alin 
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Step  3:  Assign  Weights  to  Criteria 


■  Weights  are  assigned  to  the  criteria  identified  in  Step  1  and  they  are 
prioritized  accordingly 


Criteria 

Weight 

Requirements  not  properly  defined 

0.5 

Supplier  Agreements  do  not  always  reflect 
requirements 

0.3 

Requirements  not  properly  managed 

0.2 

Alin 
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Step  4:  Pair-wise  Comparison  of  Alternatives  for  each  Criteria 


■  Using  the  AHP  method,  and  considering  the  criteria,  selected  PAs,  and 
observations,  a  prioritization  of  practices  per  criteria  is  performed. 

■  Prioritization  for  “Requirements  not  properly  defined  criteria  are  shown 
below 


R linking  for  "Requirements  not  properly  Defined" 

Practices 

Description 

Priority 

RD  SP  1.2 

Consistently  develop  and  document  MRS 

0.41 

RD  3P2.1 

Consistently  document  ERS  and  establish  link  between 

ERS  and  MRS 

0.32  | 

RD  SP3.2 

Consistently  establish  and  maintain  definition  of  required 
functionality  in  ERS 

0.1 

RD  SP1.1 

Consistently  elicit  and  document  customers  needs  and 
expectations 

0.07 

RD  SP  3.5 

Consistenlty  validate  product  requirements 

0.06 

RD  SP  3.3 

Analyze  completeness  and  sufficiency  of  requirements  ina 
consistent  fashion 

0.04 
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Step  5:  Calculate  Priority  and  Expected  Economic  Impact 


■  After  prioritizing  alternative  solutions  by  criteria,  the  method  allows  to 
prioritize  all  alternatives  analyzed. 


Practices 

Description  of  Alternatives 

Priority 

Estimated  Cost 
Reduction 

RD  SP  1.2 

Consistently  develop  and  document  MRS 

0.22 

$33,000.00 

RD  SP2.1 

Consistently  document  ERS  and  establish  link  between 

ERS  and  MRS 

0.17 

$25,500.00 

RD  SP3.2 

Consistently  establish  and  maintain  definition  of  required 
functionality  in  ERS 

0.15 

$22,500.00 

RD  SP1.1 

Consistently  elicit  and  document  customers  needs  and 
expectations 

0.11 

$16,500.00 

RD  SP  3.5 

Consistenlty  validate  product  requirements 

0.09 

$13,500.00 

RD  SP  3.3 

Analyze  completeness  and  sufficiency  of  requirements  ina 
consistent  fashion 

0.08 

$12,000.00 

RM  SP1.5 

Identify  and  document  inconsistencies  between  work 
products  and  requirements 

0.07 

$10,500.00 

SAM  SP1.2 

Select  Suppliers  based  on  their  ability  to  satisfy 
requirements 

0.05 

$7,500.00 

SAM  SP2.3 

Accept  acquired  product  verifying  it  meets  requirements 

0.03 

$4,500.00 

RM  SP1.4 

Establish  and  maintain  bi-directional  traceability 

0.02 

$3,000.00 

SAM  SP  2.1 

Review  COTS  products  to  ensure  they  satisfy  requirements 

0.01 

$1 ,500.00 

Total 

1 

$150,000.00 

Alin 


Questions  ? 


Raytheon 

Customer  Success  Is  Our  Mission 


Wasted  Days  and  Wasted  Nights 

-Leveraging  Your  Appraisal  Team  As  A  Resource  - 

November  1 6,  2005 


Timothy  Davis 

Raytheon  Missile  Systems 
Systems  Engineering  Center 
1 151  E  Hermans  Rd,  MS  805/Cl 
Tucson,  AZ  85706 
520-794-81 55 
tjdavis@raytheon.com 


Thomas  Lienhard  Jr 

Raytheon  Missile  Systems 
Software  Engineering  Center 
1 151  E  Hermans  Rd,  MS  805/K4 
Tucson,  AZ  85706 
520-794-2989 

Thomas_G_Lienhard@raytheon. 

com 


SCAMPI  -  An  Evidence  Collectors  View 


Raytheon 

Missile  Systems 


The  Scene  Opens  With  The  Clock  Nearing  Midnight 

>  Evidence  collectors  are  frantically  trying  to  finish  up  the  task  of 
putting  representative  evidence  into  the  Process  Implementation 
Indicator  database  before  the  Appraisal  Team  arrives  on  Monday. 

>One  of  the  evidence  collectors  blurts  out  “The  Appraisal  Team  has 
been  here  twice  now  and  they  keep  saying  the  same  thing:” 


“These  rocks  are  no  good,  go 
get  a  different  set  of  rocks” 
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SCAMPI  -  An  Appraisal  Team’s  View 


Raytheon 

Missile  Systems 


The  Scene  Opens  with  the  Clock  Nearing  Midnight 

>The  Appraisal  Team  is  trying  to  dig  through  the  contents  of  the  Pll  data 
base  and  trying  to  make  some  sense  of  how  the  evidence  is 
representative  of  the  process  areas  being  reviewed. 

>  A  member  of  the  Appraisal  Team  blurts  out  in  frustration: 


“When  are  these  guys  going  to  learn  to 
simply  give  us  what  we  asked  for  ???” 
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What’s  the  Problem? 


Raytheon 

Missile  Systems 


The  problem  is  the  disconnects  that  occur  between  the 
Appraisal  Team  and  the  Evidence  Collectors 


Knowledge 


Expectations 


Communication 


Disconnect  #  1  -  Knowledge 


Raytheon 

Missile  Systems 


> ATM’s  have  knowledge  of  model  and  application  to  real 
process 

-  Not  just  book  knowledge 

-  Often  has  limited  knowledge  of  Site  specific  culture  and 
program  unique  implementations 

—  May  not  have  sufficient  knowledge  on  how  to  interpret  the 
artifacts 

>  Evidence  collector  has  knowledge  of  program  and  site 
processes 

-  Not  necessarily  a  SCAMPI  Methodology  expert 

—  May  not  be  fluent  in  CMMI-ese 

—  May  not  have  sufficient  knowledge  to  interpret  the  model 
as  it  relates  to  program  artifacts 


Page  5 


Disconnect  #2  -  Expectations 


Raytheon 

Missile  Systems 


>ATM  expect  evidence  and  comments  (PHD)  to  tell  the 
story  of  how  the  programs  have  instantiated  the  process 

-Expects  to  be  in  verification,  not  discovery! 

-Expects  evidence  to  be  relevant,  concise  and  sufficient 

> Evidence  collector’s  expect  implicit  details  on  what  is 
needed  to  satisfy  the  model 

-Expects  detailed  feedback  on  SCAMPI  results 

-Expects  consistency  from  appraisal  type  to  appraisal  type 
and  from  mini-team  to  mini-team 
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Disconnect  #  3  -  Communications 


Raytheon 

Missile  Systems 


>  ATM’s  know  the  vocabulary  in  the  CMMI  and  are 
influenced  by  their  unique  experiences. 

—  Tend  to  communicate  with  model  jargon 

-  Assume  others  have  same  knowledge  or  thoughts 

>  Evidence  collector  knows  the  vocabulary  of  their 
site,  and  have  a  varying  knowledge  of  CMMI 
vocabulary. 

-  Tend  to  communicate  in  terms  of  local  or  program 
process  and  then  try  to  relate  to  CMMI  speak 

—  Do  not  instinctively  know  what  to  do  with  CMMI 
Dhrases  such  as  “For  at  least  one  program, 
nsufficient  evidence  was  provided” 
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Symptoms  of  the  “Disconnect”  Disease 


Raytheon 

Missile  Systems 


>  Wasted  calendar  time  and  dollars 
>Long  hours 

>Too  much  evidence  collected  (Quantity  not  Quality) 

>  Frustrated  ATMs,  Evidence  Collectors,  Sponsors 

>  Unnecessary  reviews 

>  Poor  appraisal  results 
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Exploring  the  Fixes 


Raytheon 

Missile  Systems 


Option  1  -  Send  everybody  to  appraisal  training 

-  Expensive$$$$$ 

-  Results  in  “Book  Learned”  knowledge 

Option  2  -  Use  previous  ATMs  as  evidence  collectors. 

-  Limited  availability  (they  have  real  jobs  too) 

-  Minimal  knowledge  carryover  into  the  appraisal 

Option  3  -  Leverage  off  the  knowledge  and  experience  of 
existing  ATMs 

-  Hmmm.  sounded  promising  !!!!! 
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Facing  Realities 


Raytheon 

Missile  Systems 


>  Need  to  develop  a  mechanism/process  to  bridge  the 
disconnects  between  the  ATM,  Evidence  collectors,  and 
project  personnel. 

>  Need  to  provide  an  environment  where  knowledge  and 
expectations  can  be  openly  shared 

>  Must  mitigate  the  typical  communication  problems  by 
eliminating  the  communication  bottlenecks. 


SHOULDER-TO-SHOULDER  REVIEWS 
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Shoulder-To-Shoulder  Reviews 


Raytheon 

Missile  Systems 


>  Current  ATM’s  working  with  EC  and  Project  personnel 
“shoulder-to-shoulder” 

>  Open,  honest,  two-way  dialog  to  hammer  out  and 
understanding  of  what  is  expected  by  the  model,  what 
the  program  produces,  and  then  what  is  missing 

—  Match  up  model  expectations/terminology  with  how 
the  program  operates  and  then  tell  the  story 

>  Sufficient  time  allocated  for  iterative  reviews  prior  to  next 
“appraisal” 

>  NOT  for  the  sponsor  or  management 

>  Care  needs  to  be  taken  to  separate  “church  and  state” 
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Results  Achieved 


Raytheon 

Missile  Systems 


>  Achieved  what  appeared  to  be  un-achievable  (based  on  the 
schedule) 

>  Reduced/Eliminated: 

-  Pre-Appraisal  Panic 

-  Levels  of  Frustration  on  all  side 

-  Wasted  calendar  time  and  dollars 

-  Quantity  of  evidence  collected 

-  Quantity  of  INFO  requests 

-  Unnecessary  reviews 

>  Improved: 

-  Model  knowledge  of  the  programs  and  EC 

-  ATM  understanding  of  the  program’s  implementation  and  issues 

-  EC  and  Project  understanding  of  ATM  issues  and  concerns. 

-  Quality  of  evidence  (not  quantity) 

-  Quality  of  the  question  and  answer  sessions. 

-  Results  of  Appraisals 


Page  12 


Shoulder-to-Shoulder  Lessons  Learned 


Raytheon 

Missile  Systems 


>  Do  not  assume  the  other  person  knows 

>S2S  outcome  is  not  a  management  presentation 

>S2S  output  must  be  given  to  programs  and  is 
focused  at  the  program  level 

>  Adequate  time  must  be  give  for  S2S,  4  hours  per 
PA  (SP’s  only)  was  our  average 

>  Golden  artifacts  aren’t  really  golden,  more  like 
bronze 

>  Iterative  S2S  with  same  people 
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Evidence  Collector  Survey  Results 


Raytheon 

Missile  Systems 


“How  affective  was  communications  between  you  and  the 
appraisal  team  before  and  after  Shoulder-to-Shoulder?” 

BEFORE: 

“It  was  done  with  written  requests  through  a  bottleneck....  The 
(ATMs)  were  those  ignorant  boobs  sequestered  from  the  real  world 
in  some  room  full  of  computers” 

AFTER: 

“At  first  I  thought  S2S  reviews  were  silly  and  tedious;  now  I  wish 
they  had  been  done  even  sooner !” 

“I  think  S2S  really  helped  us  to  clarify  what  the  Appraisal  Team  was 
looking  for... what  ana  how  they  interpreted  the  model” 

“I  would  say  that  the  S2S  is  an  indispensable  component  of  a 
successful  appraisal” 
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QUESTIONS? 


Raytheon 

Missile  Systems 
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RECP 


Customer  Success  Is  Our  Mission 


How  Has  CMMI  Improved  Our 
Program  &  Project  Performance  - 
A  Raytheon  Perspective 


John  H.  Evers,  D.Engr. 
Raytheon  Processes  PM 
1 5  November  2005 


CMMI® 

Current  Status  by  Site 


O  RSL  Harlow 
O  Northern  Ireland 


Marlboro/Sudbury  /Tewksbury 


fedford/Sudbury/ 
Tewksbury/ Andover 


^rtsmouth 


Falls  Church 


CMMI®  is  a  DoD/Industry  model  for  improvement 
defining  requirements  for  processes  using 
industry  best  practices.  Our  initial  plans  are  to 
achieve  “Level  3”  across  the  Company. 


Site  has  achieved 
w  CMMI  Level  3 

Site  has  achieved 
CMMI  Level  4/5 


®  CMMI  is  registered  in  the  U.S.  Patent  and 
Trademark  Office  by  Carnegie  Mellon  University 


Raytheon  Six  Sigma 


Customer  Success  !s  Our  Mission 


Raytheon  Process  Environment 


Capability  Maturity 
Model  Integration: 

The  yardstick  for 
judging  the  maturity  of 
our  processes 


-taytneon  six  sig 

How  we  improve  our 
processes 


Mission  Assurance: 

Where  we  define 
1 00%  customer 
success 


RECP 


Start  with  a  common  process  (IPDS),  assess  yourself 
against  the  process  using  CMMI  and  continually  improve 
the  process  using  Raytheon  Six  Sigma 
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Customer  Success  Is  Our  Mission 


Improvement  Observations 

•  Greatest  benefits  from  improved  processes  on 
programs  using  those  processes  from  the  start 


-Deployment  to  new  starts  (or  equivalent)  vital 
-Joint  stake  with  customers  to  maximize  benefits 
-Limited  to  no  benefits  to  existing  programs 


•  Many  development  programs  multi-year  efforts; 
time  needed  to  see  ultimate  benefits 


•  Primary  benefit  of  Level  3  is  stabilization  in 
processes,  increased  consistency  in  program 
execution;  greater  benefits  with  Level  4  /  5 


Benefits  take  time  to  accrue 


Raytheon’s  Position 


Customer  Success  Is  Our  Mission 


•  Committed  to  using  CMMI  as  the  primary  model 
for  process  improvement 

-Moving  towards  Level  5  across  nearly  all  businesses 
-Connected  to  our  business  objectives 


•  Seeking  improved  processes  so  we  can  deliver 
better  products  and  solutions  to  our  customers 

-Know  it’s  not  an  “overnight  journey”;  we’re  in  this  for 
the  long  haul 


Emphasis  on  Continual  Improvement 


RECP 
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Customer  Success  Is  Our  Mission 


Cnmc^ie  Mel  Ion 

Software  Engineering  Institute 


Pittsburgh,  PA  15213-3890 

Measuring  Performance: 

Evidence  about  the  Results  of  CMMI® 

Dennis  R.  Goldenson 

Diane  L.  Gibson 

Software  Engineering  Institute 


5th  Annual  CMMI  Technology  Conference  &  User  Group 
Denver;  November  2005 


Sponsored  by  the  U.S.  Department  of  Defense 
©  2005  by  Carnegie  Mellon  University 
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Cnmc^ie  Mel  Ion 

Software  Engineering  Institute 


Today’s  Talk 

Measuring  Performance:  Why  Care?  What  Counts? 

Summary  of  existing  results 

More  detailed  results:  Maturity  Levels  2  &  3 

More  detailed  results:  High  Maturity 

Current  Directions 


Capability  Maturity  Model,  Capability  Maturity  Modeling,  CMM,  and  CMMI  are  registered 
in  the  U.S.  Patent  and  Trademark  Office  by  Carnegie  Mellon  University. 
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CarnejricIVMIon 

Software  Engineering  Institute 

Why  Measure  Performance 
Outcomes  of  CMMI? 

Initially ... 

•  To  verify  that  the  there  was  value  in  beginning  to  use 
CMMI,  in  comparison  to  the  SW-CMM 

Over  the  more  recent  period  ... 

•  To  provide  ongoing  objective  evidence  about  the  value 
of  CMMI 

In  general: 

•  To  support  evidentially  based  continuous  process 
improvement 


©2005  by  Carnegie  Mellon  University 


page  3 


Carnegie  Mellon 

Software  Engineering  Institute 

Why  Objective  Evidence? 

Trustworthy  Evidence  is  Essential  for: 

•  Addressing  skepticism  about  the  benefits  of  any  model- 
based  process  improvement 

•  Demonstrating  the  value  of  CMMI  over  its  source  models 
But  also  for: 

•  Building  commitment  &  obtaining  resources 

•  Providing  input  to  improve  processes  &  technologies 

•  Comparing  results  with  other  organizations 

•  Enhancing  quantitative  management 

•  Informing  the  development  &  evolution  of  the  CMMI 
Product  Suite 


©2005  by  Carnegie  Mellon  University 
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Carnegie  Mellon 

Software  Engineering  Institute 

What  is  Legitimate  Evidence? 

Measured  performance  results  due  to: 

•  Adoption  or  upgrade  to  CMMI 

•  Systems  engineering  &  other  “new”  model  content 

•  Broadened  organizational  scope  across  disciplines 

-  Especially  integration  of  software  &  systems 

•  Maturity  or  capability  improvement  initiatives 

-  Comprehensive  or  selected  processes 

•  Improvement  in  areas  originally  defined  by  the  SW- 
CMM 

They’re  all  pertinent ...  just  different 

•  It  depends  on  your  purpose  which  ones  are  of  interest 

•  Be  careful  to  specify  your  purposes... 
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Cnrnc^ic  Mel  Ion 

Software  Engineering  Institute 

What  Does  it  Mean  to  “Implement” 
CMMI? 

An  organization’s  processes  are  not  the  same  as  model 
processes! 

•  Organizational  units  implement  &  institutionalize 
processes  for  many  reasons 

-  Often  based  on  multiple  sources  &  perspectives 

•  Processes  based  on  the  same  model  can  differ  widely 

•  Processes  are  implemented  to  achieve  different  goals  and 
outcomes 

Questions: 

•  Can  we  expect  to  find  common  measures  of  performance? 

•  When  do  we  need  the  common  measures? 


©2005  by  Carnegie  Mellon  University 
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Carnegie  Mellon 

Software  Engineering  Institute 

Are  Quantitative  Results  Enough? 

Need  more  context  to  make  the  quantitative  results 
meaningful 

•  Can  we  do  that  without  revealing  proprietary  or  other 
sensitive  information? 

Need  enough  detail  to  describe  what  was  done: 

•  What  improvements  have  been  made? 

•  Why  were  these  improvements  chosen? 

•  How  are  the  results  used? 
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Cnmc^ie  Mel  Ion 

Software  Engineering  Institute 


Today’s  Talk 

Measuring  Performance:  Why  Care?  What  Counts? 

Summary  of  existing  results 

More  detailed  results:  Maturity  Levels  2  &  3 

More  detailed  results:  High  Maturity 

Current  Directions 


Capability  Maturity  Model,  Capability  Maturity  Modeling,  CMM,  and  CMMI  are  registered 
in  the  U.S.  Patent  and  Trademark  Office  by  Carnegie  Mellon  University. 


©2005  by  Carnegie  Mellon  University 
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Carnegie  Mel  Ion 

Software  Engineering  Institute 

Reports  &  Tutorials 

Demonstrating  the  Impact  and  Benefits  of  CMMI®:  An 
Update  and  Preliminary  Results,  Special  Report, 
CMU/SEI-2003-SR-009,  Software  Engineering  Institute 
October  2003. 

Conference  presentations  &  posters 
Tutorials: 

•  Guidance  about  scoping  &  calculating  ROI  analyses 

•  Processes  &  models  for  estimating  ROI  proactively 

Benchmarking  CMMI  Cost  and  Impact:  Interim  Report, 
December  2004  (Distribution  of  full  document  limited  to 
benchmark  contributors.) 
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Carnegie  Mel  Ion 

Software  Engineering  Institute 


A  Framework  of  Costs  &  Benefits 


Process 
Capability  & 
Organizational 
Maturity 


BENEFITS 

Process 

Adherence 

Cost 

Schedule 

Productivity 

Quality 

Customer 

Satisfaction 
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CarnejricIVMIon 

Software  Engineering  Institute 

The  Documented  Results  So  Far 

Evidence  from 

•  Conference  presentations 

•  Published  papers 

•  Direct  communication  with  the  SEI 

How  Trustworthy? 

•  Public  reports  from  appraised  organizations 

•  Private  communications 

•  Information  reviewed  by  SEI  technical  staff 

Results  are  drawn  from  30  organizations 

•  Several  of  which  are  larger  enterprises  with  more  than 
one  constituent  organization 
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Carnegie  Mel  Ion 

Software  Engineering  Institute 

Performance  Results  Summary 


Improvements 

Median 

#  of  data 
points 

Low 

High 

Cost 

20% 

21 

3% 

87% 

Schedule 

37% 

19 

2% 

90% 

Productivity 

67% 

16 

11% 

255% 

Quality 

50% 

18 

29% 

1 32% 

Customer  Satisfaction 

14% 

6 

-4% 

55% 

Return  on  Investment 

4.8  :  1 

14 

2  :  1 

27.7  :  1 

•  N  =  24,  as  of  9  November  2005 

•  Organizations  with  results  expressed  as  change  over  time 
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Carnegie  Mellon 

Software  Engineering  Institute 

Existing  Measures  and  Bases  for 
Comparison  Differ:  Some  Caveats 

Performance  categories  combine  results  from  a  wide 
variety  of  cases 

-  Ranging  from  pilot  projects  about  the  effects  of  particular 
processes 

-  To  organization-wide  improvement  initiatives  covering  the 
full  scope  of  CMMI 

Other  factors  sometimes  occur  simultaneously  with  CMMI- 
based  process  improvement 

-  E.g.,  reuse,  personnel  changes  or  new  technologies 

Specific  measures  differ  as  well 

-  E.g.,  total  cost  or  cycle  time  versus  discrepancies  between 
estimates  &  actuals 
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CarnqricIVMIon 

Software  Engineering  Institute 

However... 

Valid  &  valuable  comparisons  can  be  made 

•  So  long  as  we  distinguish  properly  among  them 

Varied  cases  &  measures  provide  ample  proof  of  concept 

•  About  the  potential  of  CMMI-based  process  improvement 

The  same  results  may  not  always  be  repeatable  elsewhere 

•  But,  we  often  see  very  impressive  performance  effects 

•  The  probability  is  very  low  that  such  results  are  due  to 
chance  alone 

Multi  trait,  multi  method 
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Carnegie  Mellon 

Software  Engineering  Institute 

Variety  in  measures:  Cost 

Measures  included: 


Reductions  in 

costs 

cost  of  quality 
poor  quality 
costs  of  rework 

cost  of  delivery 
accuracy 

defect  find  &  fix  costs 
variation  in  CPI 
overhead  rate 
software  unit  costs 
#/cost  of  process  staff 


Savings  in/due  to 

implementing  hardware 
processes 

Improved 

budget  estimation 

average  CPI 
cost  variance 
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Carnegie  Mellon 

Software  Engineering  Institute 

Variety  in  measures:  Schedule 

Measures  included: 


Reductions  in 

variation  in  schedule 
performance  index  (SPI) 

#  of  days  late 
days  variance  from  plan 
slippage  of  project  delivery 
schedule  variance 


I  m  p  r  o  ved/i  n  c  reased 

average  (SPI) 
estimation  accuracy  (III) 
schedule  variance  (II) 

%  of  milestones  met 
cycle  time  (II) 
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Carnegie  Mellon 

Software  Engineering  Institute 

Variety  in  measures:  More 
Examples., 

Productivity  measured  in,  for  example: 

-  ELOC  per  labor  hour 

-  function  point  per  FTE 

-  source  statements  per  month 

-  testing 

-  #  of  releases  per  year 

-  comparisons  between  builds 

-  software  production,  in  general 

Quality  measured  in 

-  Defects  (different  products,  stages  of  the  life  cycle) 
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Carnegie  Mellon 

Software  Engineering  Institute 

Variety  in  measures:  More 

Examples2 

Customer  Satisfaction  measured  with 

-  Award  fees 

-  Ratings 


-  Defects  avoided 

-  Post-release  defects  avoided 

-  Automation 

-  Quality  activities 

-  Process  Improvement  in  general 

-  Maturity  Level,  in  general 
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Carnqric  Mel  Ion 
Software  Engineering  Institute 

CMMI  Performance  Results  Web  Site-, 

Results  by:  http://www.sei.cmu.edu/cnnnni/results.html 

•  Performance  category  &  organization 

•  Brief  statements  &  graphical  examples 

•  Full  source  documents 


(jirnoirk'  Vic]  Ini* 

Software  Engineering  Institute 


Hama  Saarafo  Contact  Us  Site  Map  HflfiaS 's  Atow 


improving 

management 

practices 


MA  f\iA  GEilTE/V  T 


About  MaaiageTnsnl  Engineering  Acquisition  WorkwilliUe  ProilucLi?  Publications 
Ute'SEI  antJ  Services 


CIvlMi r  Performance  Results 


CMMI  Main 
Page 

°  General 
°  Models 
Q  Adoption 
°  Training 
°  Appraisals 

*  Performance 
Results 

°  Background 

°  Frequently 
Asked 
Questions 


Objective  and  Scope  |  Results  |  Providing  Results 

Objective  ami  Scope 

There  is  a  widespread  demand  for  credible,  quantitative 
evidence  about  the  results  of  process  improvement  based 
on  CMMI  models.  The  results  presented  here  are  from 
publicly  available  conference  presentations,  published 
papers,  and  individual  collaborations  with  the  SEI. 

Together,  these  results  provide  proof  of  concept  about  the  potential  of 
CMMI-based  process  improvement.  The  results  show  that  CMMI  often  leads  to 
very  impressive  improvements  in  product  quality,  project  performance,  and 
organizational  performance;  however,  the  individual  results  presented  here  may 
not  be  repeatable  in  every  organization. 


CMMI 

y 
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Carne^cMellon 
Software  Engineering  Institute 

CMMI  Performance  Results  Web  Site2 


View  by  Organization 

The  performance  results  examples  contain  brief  assertion  statements  and  their 
sources  and  sometimes  are  accompanied  by  graphic  illustrations.  To  view  all 

|(  examples  for  an  organization,  click  the  name  of  the  organization. 

1  Organization 

Accenture 

Motorola  Global  Software  Group 

Anonymous  1 

NCR 

Anonymous  2 

Northrop  Grumman  IT,  Defense  Enterprise  Solutions 

DEI  Systems  GambH 

Raytheon  Corporation,  Anonymous  site 

Fire  Support  Software  Enqineerinq  Division 

Raytheon  Network  Centric  Systems 

General  Dynamics  Advanced  Information  Systems 

Raytheon  North  Texas  Software  Enqineerinq 

General  Motors 

Reuters 

Harris  Corporation 

SAIC  System  and  Network  Solutions  Group 

IBM  Australia  Application  Manaqement  Services 

Siemens  Information  Systems  Ltd. 

JPMorqan  Chase 

Systematic  Software  Enqineerinq 

Lockheed  Martin  Corporation 

TATA  Consultancy  Services 

Lockheed  Martin  Manaqement  and  Data  Systems 

Thales  Air  Traffic  Manaqement 

Lockheed  Martin  Maritime  Svstems  &  Sensors  -  Undersea  Svstems 

The  Boeinq  Company 

Warner  Robins 

Lockheed  Martin  Maritime  Svstems  and  Sensors  -  Radar  Svstems 

Lockheed  Martin  Maritime  Systems  and  Sensors  -  Syracuse 

Lockheed  Martin  Systems  Inteqration 
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Carnegie  Mol  Ion 

Software  Engineering  Institute 

CMMI  Performance  Results  Web  Site3 

CMMI'  Performance  Results 


IBM  Australia  Application  Management  Services 

The  performance  results  examples  contain  brief  assertion  statements  and  their 
sources  and  sometimes  are  accompanied  by  graphic  illustrations.  To  view  the 
graphic  or  source  for  a  statement,  click  the  View  link. 

Cost  |  Schedule  |  Productivity  |  Quality  |  Customer  Satisfaction  | 


Cost 

Assertion  Statement 


On-budget  delivery  improved  from  over  90  percent  to  nearly 
100  percent  as  the  organization  moved  from  SW-CMM 
maturity  level  3  to  CMMI  maturity  Level  5 

top  A, 


Schedule 

Assertion  Statement 


On-time  delivery  remained  well  over  90  percent,  with  a  slight 
improvement,  as  the  organization  moved  from  SW-CMM 
maturity  level  3  to  CMMI  maturity  level  5 

top  A. 


Productivity 

Assertion  Statement 


$99  million  dollars  saved  in  development  costs  due  to 
increased  productivity  as  the  organization  moved  from 
SW-CMM  maturity  level  3  toward  CMMI  maturity  level  5 


Quality 

Assertion  Statement 


40  percent  reduction  in  all  production  problems  as  the 
organization  moved  from  SW-CMM  maturity  level  3  toward 
CMMI  maturity  level  5 

On  average,  over  95  percent  of  problems  were  closed 
monthly  within  the  customer-specified  time  frame  after  the 
organization  achieved  CMMI  maturity  level  5 
Over  80  percent  reduction  in  Severity  1  problems  as  the 
organization  moved  from  SW-CMM  maturity  level  3  toward 
CMMI  maturity  level  5 

top  JL 


Customer  Satisfaction 

Assertion  Statement 


Customer  satisfaction  remained  well  over  30  percent  after 
the  organization  achieved  CMMI  maturity  level  5 

top  A, 


$103  million  dollars  saved  in  maintenance  costs  due  to 
increased  productivity  as  the  organization  moved  from 
SW-CMM  maturity  level  3  toward  CMMI  maturity  level  5 
Over  20  percent  improvement  in  account  productivity  as  the 
organization  moved  from  SW-CMM  maturity  level  3  toward 
CMMI  maturity  level  5 


Example 


©2005  by  Carnegie  Mellon  University 


page  21 


Giirnept*  Mellon 
Software  Engineering 
Institute 

CMMI 

Performance 
Results 
Web  Site, 


CMMI  ■  Performance  Results 


Assertion  Statement  Detail 


Statement 


Over  20  percent  improvement  in  account  productivity  as  the  organization  moved  from  SW-CMM 
maturity  level  3  toward  CMMI  maturity  level  5 


Organization 


IBM  Australia  Application  Management  Services 


Graphic 


Performance  Measure:  Productivity 


Account  Productivity 
(FPFTE) 


HgUFB  8:  A  ccount  Productivity  f FP/FTE } 


Uahii-nv  i  p. rtt 

Mtmtian 

dOUuO  uy  sti 


IBM  Australia  Application  Management  Services.  Software  Process  improvement  Journey:  IBM 
Australia  Application  Management  Services.  AReportfromthe  Winner  of  the  2004  Software 
™  Process  Achievement  Award  (CMU/SEI-2005-TR-002).  Nichols,  Robyn  &  Connaughton,  Colin. 
Pittsburgh,  PA:  Software  Engineering  Institute,  Carnegie  Mellon  University,  2005. 
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Today’s  Talk 

Measuring  Performance:  Why  Care?  What  Counts? 

Summary  of  existing  results 

More  detailed  results:  Maturity  Levels  2  &  3 

More  detailed  results:  High  Maturity 

Current  Directions 


Capability  Maturity  Model,  Capability  Maturity  Modeling,  CMM,  and  CMMI  are  registered 
in  the  U.S.  Patent  and  Trademark  Office  by  Carnegie  Mellon  University. 
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Statements,  Organizations  at  ML  2  &  31 

Costs 

•  Reduced  rework  costs  by  42%  at  CMMI  Maturity  Level  3 
(Raytheon) 

•  $2.1  Million  in  savings  in  hardware  engineering  processes  in 
an  organization  moving  towards  CMMI  maturity  level  3 
(Anonymous) 

•  From  a  1999  baseline  prior  to  improvement,  costs  dropped 
48%  by  2003,  as  the  organization  moved  toward  CMMI  ML3. 
(DB  Systems  GamBH) 


CMMI 
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Performance  Measure:  Cost 

Costs  dropped  48  percent  from  a  baseline  prior  to  SW-CMM 
ML2  as  the  organization  moved  toward  CMMI  ML3 

Return  On  Quality  Die  Bahn  1DBI 


Cost  /  Cost  I  Function  based  on  reliable 


1999  2000  2001  2002  2003 


Maturity 

CMMI  Level  3 


CMMI  Level  2 


Cost  reduction  achieved 
by  process  Improvement, 
substitution  of  external 
sources  and  optimized 
infrastructure 


DB  Systems.  Dr.  Alfred  Richter 


Febfuar  2D04 


DB 

Systems 

GamBH 

page  25 


©  2005  by  Carnegie  Mellon  University 


Carnegie  Mel  Ion 

Software  Engineering  Institute 


Statements,  Organizations  at  ML  2  &  32 


Schedule 

•  Percentage  of  milestones  met  improved  from  approximately  50% 
to  approximately  85%  following  organization  focus  on  CMMI 
(General  Motors) 


•  Average  variance  from  development  plan  reduced  from 
approximately  60  days  to  less  than  20  days  one  year  after 
reaching  CMMI  Maturity  Level  2  (NCR) 

•  Reduced  schedule  variance  over  20  percent  in  an  organization 
moving  towards  CMMI  maturity  level  3  (Anonymous) 

•  Increased  through-put  resulting  in  more  releases  per  year  at 
CMMI  maturity  level  3  (JP  Morgan  Chase) 


•  Achieved  95  percent  on  time  delivery  in  an  organization  moving 
towards  CMMI  maturity  level  3  (Anonymous) 
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CMMI 


Performance  Measure:  Schedule 


Some  Results  to  date 

Development  Variance  Improvement 


Carnegie  Mel  Ion 

Software  Engineering  Institute 

Statements,  Organizations  ML  2  &  33 

Productivity 

•  Used  Measurement  &  Analysis  Process  Area  to  realize  an  1 1 
percent  increase  in  productivity,  corresponding  to  $4.4M  in 
additional  value  (Anonymous) 

Quality 

•  Reduction  in  number  and  severity  of  post  release  defects  at 
CMMI  ML2  (Anonymous) 

•  More  than  80%  drop  in  defects  in  6  months  after  achieving 
CMMI  Maturity  Level  (JP  Morgan  Chase) 

•  44%  defect  reduction  following  causal  analysis  cycle  at  an 
organization  moving  towards  CMMI  maturity  level  3 
(Anonymous) 
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Performance  Measure:  Quality 

Asia  Treasury  and  Credit  Rates  achieved  CMMI  level  2  at  the  end  of 
2003.  In  the  subsequent  6  months  their  average  number  of  UAT  & 
production  defects  dropped  by  more  than  80%  (18  projects) 
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Statements,  Organizations  at  ML  2  &  34 

ROI 


•  Used  Measurement  &  Analysis  Process  Area  to  realize  a 
2.5:1  ROI  over  1st  year,  with  benefits  amortized  over  less 
than  6  months  (Anonymous) 


Process  Adherence* 

•  Marked  improvements  in  work  product  completion  after 
new  training  instituted  on  the  way  to  CMMI  Maturity  Level  3 
(CMS  Information  Systems,  Inc.) 


*  Evidence  of  this  kind  is  crucial  for  a  better  understanding  how  process 
changes  have  been  implemented.  We  have  seen  very  little  so  far: 
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Progress  during  PI  Effort 


Work  Products  Completion 


-  Early  Ranning 

PP 

—A- 

PMC 

Engineering 

-Support 

33THMS 

Information  Service sr  tnc. 
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Today’s  Talk 

Measuring  Performance:  Why  Care?  What  Counts? 

Summary  of  existing  results 

More  detailed  results:  Maturity  Levels  2  &  3 

More  detailed  results:  High  Maturity 

Current  Directions 


Capability  Maturity  Model,  Capability  Maturity  Modeling,  CMM,  and  CMMI  are  registered 
in  the  U.S.  Patent  and  Trademark  Office  by  Carnegie  Mellon  University. 
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Performance  Measure:  Cost 


IBM  Australia  Application  Management  Services 

On-Budget  Delivery 


100.0% 


80.0% 


60.0% 

40.0% 


20.0% 

0.0% 


n On-Budget  Delivery 


Figure  6:  On-Budget  Delivery 


Maturity  Level 
Notation 
added  by  SEI 
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Performance  Measure:  Schedule 


89%  of  all  deliveries  in  2003  were  on  time 


Deliveries  on  time 
66%  in  2001 
79%  in  2002 
89%  in  2003 

In  2004  we  expect  to 
fulfill  our  objective 
that  we  deliver  at  least 
90%  on  time 


5 


2002  CMM  Level  3 

2004  CMMI  Maturity  Level  4 
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CMMI 


Performance  Measure:  Productivity 


An  Employee-Owned  Company 


And  More  Trends ... 


Labor  productivity  averages  have  increased,  influenced 
by  variables  such  as  programming  languages,  technical 

i^ipi*0^®i^l®i^tS,  etC.  System  and  NetAorV;  Solutions  Group  (SNSG1 
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Performance  Measures: 

Cost,  Schedule  &  Productivity 


Comparison  of  B1  and  B2/3/4  Metrics 

NETWORK.  CENTRIC  SYSTEMS 


20 


Productivity 

B1  SIL  l&T  Productivity 
B 2/3/4  SIL  l&T  Productivity 
—  62%  improvement 


=  2.1  LOC/Hr 
=  3.4  LOC/Hr 


Ollier  Factors:  Team  gained 


J  e^r  r  I'  .  . ; 


experience  ii?  alt  aspect  of 


CPI  and  SPI 

JUL  2001  Cum  CPI/SPi 
JAN  2002  Cum  CPI/SP! 
—  5%  /  6%  Improvement 


=  .91  /  .93 
=  .96  /  .99 


@  2M3  Rs-ihMn  Compiry 
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Performance  Measure:  Quality 

Defect  prevention  using  PSP  and  CAR  at  CMMI  ML5 


O  c 

o  6 

_l 

5 

<0  o 

o 

0)  4 
a> 

1  3 

*— » 

!  2 

2  “i 

O  1 
0 

§  0 


DP  1 
CAR 


DP  2 
CAR 


DP  3 
CAR 


2  3 

Build 


6.6 

6.1 

3.5 

3.9 

2.1 

T 

[ 

Integrating  PSPsm  and  CMMI®  Level  5.  Gabriel  Hoffman,  Northrop  Grumman  IT  .  May  1 , 
2003. 
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Northrop  Grumman  IT 

Hours  invested:  124 

•  Team  training:  48 

•  Conducting  DP/CAR  Cycles:  76 

Defects  avoided:  110 

•  If  defect  density  had  remained  at  Build  1  baseline 

Hours  saved:  1650  hours 

•  At  an  estimated  cost  of  1 5  hours  per  defect 

Return: 

•  Hours:  1650/124 

•  ROI:  -13:1 
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Performance  Measure:  ROI 
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Remember 

Don’t  over  interpret  these  results  out  of  context 

•  The  cases  differ  in: 

-  Organization  &  model  scope  of  their  process  changes 

-  The  time  span  of  the  process  or  other  technology 
interventions  they  report 

-  The  specific  measures  they  use 

-  Measures  of  organizational  context 

•  Some  of  the  results  also  may  be  atypical  &  exemplary 
However 

•  They  do  constitute  ample  proof  of  concept  of  the  potential 
of  model-based  process  improvement 
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Today’s  Talk 

Measuring  Performance:  Why  Care?  What  Counts? 

Summary  of  existing  results 

More  detailed  results:  Maturity  Levels  2  &  3 

More  detailed  results:  High  Maturity 

Current  Directions 


Capability  Maturity  Model,  Capability  Maturity  Modeling,  CMM,  and  CMMI  are  registered 
in  the  U.S.  Patent  and  Trademark  Office  by  Carnegie  Mellon  University. 
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Enhancing  Quality  &  Quantity  of  the 
Evidence 

More  &  better  case  studies  are  not  enough 

•  Broadly  based  samples  needed  to  attribute  results  to 
CMMI  based  processes  versus  other  factors  / 
unintended  measurement  effects 

Need  for  a  viable  benchmarking  infrastructure  & 
community  of  practice 

•  In  a  field  where  people  aren't  comfortable  sharing 
information 
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What’s  Needed? 

Evidence  from  case  studies  can  be  accused  of  “cherry 
picking”  --  Fairly  or  not 

Must  be  proactive:  credible  comparative  evidence  is 
sorely  needed 

•  To  better  demonstrate  the  statistical  relationships 
between  process  capability  &  program  performance 

•  Controlling  for  other  characteristics  that  may  affect  both 

By  now  are  many  Maturity  Level  4  &  5  organizations 

•  Over  1 1 0,  mostly  ML5  at  this  time  last  year 

Many  CMMI  Maturity  Level  2  organizations  should  have  at 
least  selected  amounts  of  pertinent  measured  results  as 
part  of  their  PP  &  PMC  activities 
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Some  Results  of  Adopting  CMMI 


N  =  68;  Mostly  high  maturity  organizations 

Source:  Benchmarking  CMMI  Cost  and  Impact:  Interim  Report,  December  2004 
(Distribution  of  full  document  limited  to  benchmark  contributors.) 
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Results  From... 

Simple  benchmarking  exercise  presented  at  4th  CMMI 
Technology  Conference  in  Denver  last  November 

•  Focus  on: 

-  Costs  &  investment  in  process  improvement 

-  CMMI  adoption 

-  Implementation  &  appraisal  strategies 

•  A  little  on  benefits  of  CMMI-based  process  improvement 

Mostly  high  maturity  organizations 

•  Still,  quite  promising 

•  73%  have  quantitatively  measured  improvement  results 

•  68%  have  done  ROI  or  related  cost  benefit  analyses 

•  Accompanied  by  compelling  qualitative  descriptions! 
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What’s  Next? 

CMMI  performance  results  web  site 

•  Updates  &  enhancements 

A  new  summary  TR 

•  Addition  of  brief  case  reviews  (“vignettes”) 

-  To  provide  context  for  the  quantitative  results 

Articles  on  CMMI  performance  results 

•  For  Software  Process  Improvement  and  Practice 

Any  information  you  can  share  with  us  will  be 
welcomed  and  appreciated 
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What  Else? 

Enterprise  performance  measurement  and  benchmarking 

•  Focus  on  causal  analysis  of  variation  in  program 
success  and  failure 

•  Working  with  organizations  that  already  have  or  are 
willing  to  develop  common  measures 

Exploring  several  options  for  emphasis  in  FY06-07,  e.g.: 

•  A  web  based  benchmarking  service 

Perhaps  seeded  by  a  proactive  survey 

•  Focused  custom  surveys 

Any  ideas  or  information  you  can  share  with 
us  will  be  welcomed  and  appreciated 
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To  Summarize... 

There  is  ample  evidence  about  the  results  of  model-based 
process  improvement 

Still,  we  need  more  &  better  evidence 

•  Serious  attention  to  benchmarking 

-  Better  understanding  the  state  of  the  practice 

-  Understanding  what  accounts  for  relative  failure  as  well  as 
success 

•  Richer  case  studies 

•  Practical  guidance 

-  Validating  estimates  and  improving  ROI  &  process  models 

-  Measurement,  validation,  data  quality  &  analytic  methods 

Our  bottom  line:  Actionable  guidance  using  measurement  to 
inform  better  decisions 
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For  more  information  or  to  discuss  participation,  contact: 


Dennis  R.  Goldenson 
dq@sei.cmu.edu 

Diane  L.  Gibson 
dlq@sei.cmu.edu 

Software  Engineering  Institute 
Carnegie  Mellon  University 
Pittsburgh,  PA  15213-3890 
U.S.A. 
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Purpose  of 
Presentation 


▲  To  shed  light  on... 

▼  Today’s  software  market  &  how  it’s  different  from 
the  software  market  when  QA  [process  discipline] 
was  first  applied. 

▼  Why  traditional  QA  fails  in  today’s  software 
development  environment. 

▼  How  QA  needs  to  be  structured  to  work  in  today’s 
software  development  environment. 

▼  What  “agile”  software  development  is  and  isn’t. 

▼  How  agile  software  development  can  be  disciplined. 


Translating  technology  dollars  into  business  sense.SM 
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▲  Introduction 

▲  Understanding  Agile 

▲  Role  and  Goal  of  “QA” 

▲  Historical  Role  of  QA 

▲  QA  in  Business 

▲  A  Word  About 
Development 
Processes 
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▲  QA  in  the  Context  of 
Development 
Processes 

▲  Rethinking  the  Quality 
Abstraction 

▲  An  Implementation 
Example  with  Scrum 

▲  Conclusion 
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Introduction 


Technolosy  ’ 


A  Why  “Re-Think”  the  Quality  Abstraction? 

A  QA’s  Legacy  Mindset 
A  Software  Today  and  Yesterday 
A  Movement  in  the  Software  Industry 
A  WhyAA/here  Lightweight  and  Heavyweight  Collide 
A  Disciplined  vs.  Undisciplined 


A  QA  as  a  Valuable  Asset 


Translatinq  technoloqy  dollars  into  business  senses 


Why  “Re-Think”  The 
Quality  Abstraction? 


The  Technology 
Strategy  Companysm 


Align  process  and  product  technologies 


▲  Align  development  environment 


Align  with  market  forces 


Tech  nolost  ’ 
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QA’s  Legacy  Mindset 


Large  Products 


▲  Software  a  component  of  the  product 


Technology  trade-off 


Software  is  now  the  entire  product 


Technolost  1 
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Software  Today  and 
Yesterday 
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Techno  lost  ’ 


▲  Compare  today’s  software  products  to  those  1 0, 
15,  2Q+  years  ago 

▲  How  are  they  similar? 

▲  They  are  more  different  than  they  are  similar. 

▲  QA  hasn’t  changed  with  the  technologies  and 
methodologies 
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▲  Companies  attempting  process  QA  initiatives  found 
everywhere  on  the  side  of  the  road 

▲  Agile/Lightweight  seen  as  a  “way  out” 

▲  Cannot  ignore  the  trend 

▲  “Lightweight”  in  response  to  “Heavyweight” 
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Heavy  Collide 

▲  Attitude 
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▲  Strengths 


Weaknesses 


Typical  approach  to  QA  propagates  legacy  methods 
&  mindset 


There’s  no  such  thing  as  robust  QA  in  lightweight 
development  ?! 
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Disciplined  vs. 
Undisciplined 
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▲  Lightweight  is  thought  of  as  undisciplined? 

▲  Could  it  be  QA’s  fault? 

▲  Can  lightweight/agile  development  also  be  robust? 

▲  Can  QA  become  appropriately  agile? 

▲  Can  a  mindset  “re-set”  about  QA  be  applied? 

▲  Could  an  abstraction  be  created  for  QA  that  works 
in  any  environment? 

▲  Could  it  be  used  to  improve  QA  in  non-lightweight 
software  environments? 
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QA  activities  are  expected  to  be: 

▼  value-added 

T  a  component  of  a  comprehensive  product  development 
process. 

Look  at  QA  in  terms  of: 

T  its  basic  goals. 

▼  how  to  adapt  what  QA  professionals  do  to  meet  those  goals. 

Developed  a  QA  approach  that: 

T  works  in  any  environment 

T  Is  still  in  complete  compliance  with  standards,  and  policies. 

A  change  in  abstraction  will  cause: 

▼  how  QA  "shows  up”  on  a  project, 

T  not  what  QA  is  expected  to  accomplish. 
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▲  Lightweight  development  varies  widely  from 
organization  to  organization. 

▲  Lightweight/ Agile  the  reputation  as  undisciplined. 

▲  Narrow  implementations  of  the  concepts,  rarely 
following  any  formal  development  guidelines. 

▼  .-.  the  reputation  is  unfair. 

▲  Coding  without  rules,  process  discipline,  or 
management  tools  is  undisciplined. 

▼  This  is  not  what  agile  development  is. 

▼  Any  more  than  the  original  intent  of  effective  QA 
was  to  be  heavy-handed. 
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The  purpose  of  lightweight  development  is  to  allow 
for  better  productivity. 

The  enemy  of  productivity  is  heavy-handed  process 
controls. 

True,  some  developers  pursue  lightweight 
development  thinking  they  can  shed  controls, 
checks,  and  balances  necessary  to  make  good 
products. 

This  is  far  from  what  lightweight  is  about. 
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▲  If  this  were  true,  then  lightweight  developers  would 
be  operating  under  a  modus  operandi  that  reads: 

▼  “produce  quality  software  in  the  absence  of  any 
process”. 

>  This  would  be  absurd. 

>  Lightweight  supporters  do  not  agree  with  this. 

▲  It’s  not  the  absence  of  process  that  makes  a 
development  method  lightweight. 

▲  It’s  the  absence  of  unnecessary  or  obstructive 
processes  that  makes  a  method  lightweight. 
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Agile  Development 
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The  minimum,  most  unobtrusive  approach  to 
developing  software  that  produces  a  quality  product 
when  the  customer  expects  to  get  it,  at  the  price 
they  expect  to  pay. 
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Agile  Alliance 


▲  Principles: 

▼  Satisfy  the  Customer 

thru  valuable  software 

▼  Changes  happen,  harness 
them  for  the  customer’s 
benefit 

▼  Deliver  working  product 
frequently 

▼  The  business  must  work 
with  the  developers 

▼  Hire  motivated  people, 
support  them,  let  them 
work 

▼  Face-to-face  beats  paper 
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▼  Working  software  is  the  best 

measure  of  progress 

▼  If  it’s  not  sustainable,  it’s  not 
agile 

▼  Agility  depends  on  continuous 
attention  to  technical 
excellence  &  good  design 

▼  Simplicity  is  key  to  maximizing 
work  not  done 

▼  Self-organizing  teams  produce 
the  best  technical  results 

▼  Regularly  reflect  on  becoming 
more  e  ffective  and  tune  &  adjust. 
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Agile  Alliance,  2 

▲  Manifesto: 

“We  value: 

Individuals  and  interactions  over  processes  and  tools 
Working  software  over  comprehensive  documentation 
Customer  collaboration  over  contract  negotiation 
Responding  to  change  over  following  a  plan 


That  is,  while  there  is  value  in  the  items  on  the  right, 
we  value  the  items  on  the  left  more.  ” 

Is  this  anti-process? 

Can  anyone  prefer  the  process  to  actually  delivering  the 
product? 
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▲  How  to  Entice  Developers  to  Follow  a  Process 


Working  Definition  of  QA 


▲  Working  Policy  Statement  of  QA 
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Realistically,  in  the  typical  QA  approach  not  all 
activities  performed  to  “satisfy  QA  requirements” 

▼  productive, 

▼  pro-active, 

▼  value-added  contributions  to  producing  the  product. 

Otherwise,  developers  would  use  typical  QA 
processes. 
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Time  away  from  development  isn’t  productive 
▼  [documentation  of  work  already  performed] 

Reconciling  heavyweight  and  lightweight  practices 
will  be  found  by  bridging  this  gap. 

Create  processes  that  parallel  development. 
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A  process/effort  that  ensures  that 
processes  are  followed,  that 

the  processes  have  us  doing 

Y  the  right  things, 

▼  the  right  way,  and 

when  they  fail  to  be  used  or  fait  to  perform 
as  expected  we  have  a  way  to 

Y  correct, 

Y  adjust,  or 

Y  escalate  the  matter  until  it  is  resolved  to 
everyone’s  satisfaction. 
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All  processes  must  forge  e  working  relationship  which 

T  actively  supports  development’s  productive  activities 

T  avoids  creating  additional  effort  for  development  functions 
outside  of  the  project’s  stated  development  function  processes 

▼  designs  processes  in  collaboration  with  the  project’s 
development  community 

▼  allows  the  process  owners  to  achieve  their  process  and 
product  oriented  objectives 

▼  reaches  consensus  on  a  balance  between  process  and 
productivity. 

The  goal: 

T  fully  integrate  the  necessary  process  steps  into  activities  that 
add  value  to  the  development  effort  while 

T  resulting  in  insight,  predictability,  measurements  and 
traceability  of  process  effectiveness. 
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▲  QA  undergoes  continuous  improvement  in  terms  of 
its  application  as  well  as  acceptance. 

▲  QA  is  an  official  component  of  many  project  plans 
and  a  valued  resource  in  many  projects  and 
organizations. 

▲  QA  can  hold  up  a  project  with  process  problems, 
and  “dress  down”  a  project  manager  for  skipping 
steps. 
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▲  Unfortunately,  QA  is  still  sidelined  too  often  when 
business  needs  take  priority. 

A  Too  often,  QA  still  has  the  reputation  of  “policing” 
rather  than  a  contributing  to  the  effort. 

A  Frequently,  business  owners  will  bypass  managers 
and  go  directly  to  developers  when  such  layers  are 
seen  as  getting  in  the  way. 
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▲  Showing  the  business  value  of  QA  through  analyses, 
6-sigma  SPC,  and  other  techniques  are  still  more 
reactive  than  pre-active. 

▲  What  developers  [and  executives]  want  are 
processes  that  implement  QA  so  that  they  don’t 

▼  slow  progress, 

▼  break  momentum,  or 

▼  install  the  sense  that  people  are  being  policed. 

▲  Such  processes  are  demoralizing. 
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▲  Processes  are  fueled  by  people 

▲  People  hate  heavy-handed  processes. 

▲  Developers  seek  “lightweight”  methods  in  hopes  of 
finding  refuge  from  heavy-handed  processes. 

▲  Many  throw  out  the  mantle  of  all  processes. 

A  The  origins  of  QA  standards  explains  much . 
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▲  Early  software  projects  were 

▼  big, 

▼  slow,  and 

▼  geographically  dispersed 

▲  Early  projects  were  characterized  by 

▼  layers  of  bureaucracy 

▼  designed  around  project  management  methods  that 
also  built  tanks,  planes,  and  ships. 


▲  Based  on  manufacturing  work-flow  and  controls. 
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▲  These  QA  methods  fail  to  achieve  their  intended 
goals. 

▲  Software  development  paradigm  shares  very  little 
with  the  manufacturing  paradigm. 

▲  Methods  of  performing  QA  have  not  made  the  shift 
across  the  industry. 

▲  Defense  and  similar  large-scale  old-style  projects 
shaped  much  of  what  is  known  today  about  QA. 

▲  Compared  to  today’s  technologies  and  the  speed  to 
market,  these  legacy  projects  provide  a  very 
limiting  pool  of  experience. 
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QA 

Manual 


V 


ISO-9000 


sM 


Mil-Q-9858A 


V 


CMMI® 


Are  so  many  businesses  and 
projects  so  similar  that  they  can  all 

use  that  same  QA  approach?  DoD-STD-2168 
Of  course  not! 


QA 

Manual 
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▲  The  non-productive  activities  and  paperwork 
created  by  legacy  QA  abstractions  is  most  felt  at 
the  development  level. 

▲  In  many  large,  complex  projects,  the  additional 
effort  and  time  needed  to  follow  the  processes  are 
easily  absorbed  by  the  project. 

▲  The  pace  of  these  projects  are  such  that  the 
deliberate  [if  not  judicious]  addition  of  time  and  work 
can  be  handled. 
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Development  needs  to: 

▼  stay  productive, 

▼  control  costs,  and 

▼  keep  people  motivated. 

The  effort  to  follow  the  process  should  not 
overshadow  the  pace  of  the  project. 

Agile  development  recognizes  the  need  for 
processes  that  allow  a  project  to  get  done  at  the 
pace  of  the  project. 

Many  processes,  QA  included,  have  fallen  short 
because  they  do  not  account  for  the  pace  and 
complexity  of  the  project. 
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▲  Modern  software  needs  QA  processes: 

▼  more  closely  fitted  to  each  project, 

▼  dynamically  adapting  to  the  project  and  making 
development  cheaper,  better,  and  faster  on  every 
subsequent  project. 

A  Truly  add  business  value  to  the  QA  process. 

A  On  time  working  product  is  a  must. 

A  Processes  must  reflect  the  demands  of  the 
customer.  First  and  foremost. 

A  Processes  must  be  adaptive  and  scalable  to  handle 
exceptions. 
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▲  Historically,  legacy  QA  processes  not  designed  with 
attention  to  business  goals. 

▲  Latest  models  promote  processes  that  add  value. 

▲  Few  implementations  ever  achieve  that. 

▲  Instead,  companies  supplement  existing  processes 
with  a  disruptive,  paper-intensive  meta-layer. 

▲  Produce  evidence  that  a  process  is  being  followed. 

▲  Do  not  contribute  to  productivity. 

A  This  has  not  changed  in  decades. 
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▲  Hardware  can  be  designed  and  manufactured  in  any 
one  of  several  ways: 

▼  Design  can  be  on  paper,  or  using  Computer  Aided 
Design  [CAD]  systems. 

▼  Manufacturing  can  be  by  skilled  artisans  or  can 
employ  automated  systems. 

A  These  are  the  “development  methodologies”. 

A  Tools  and  tool  control,  inspection,  inventory  control, 
materials  ordering,  environmental  controls, 
organizational  needs. 

A  These  are  “management  methodologies”. 
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The  development  and  management  methodologies, 
therefore,  are  distinct. 

Not  completely  de-coupled,  however  one  does  not 
dictate  the  other. 


▲  They  must: 

▼  complement  and  support  one  another. 

▼  work  together  to  achieve  business  goals. 

▲  Desirable  to  be  optimized  to  work  in  the  same 
business  and  operations  strategy  models. 

▲  Fundamentally,  whether  blueprints  are  drawn  by 
hand  or  by  CAD  is  not  dictated  by  how  the  flow  of 
material  is  controlled  through  the  plant. 
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▲  In  the  software  world,  for  example,  CMMI®  doesn’t 
care  what  development  methodology  is  used. 

▲  CMMI®  doesn’t  dictate  use  of  the  “Waterfall”  or 
Spiral  models,  or  imply  that  XP  is  better  than 
Scrum,  “Crystal  Light”,  and  so  on. 


Distinguishing  the  software  development 
methodology  from  the  software  management 
methodology  eliminates  one  of  the  barriers  to 
managing  QA  in  lightweight  development 
environments. 
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▲  QA’s  role  in  the  development  process  is  fairly 
simple. 

▲  Standards,  methods  and  processes  need  to  be 
followed  and  need  to  work  well  for  the  project. 
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Development  Process 


QA  is  responsible  for  ensuring: 

▼  project’s  methodologies  are  taught  to  new  developers  on  the 
project; 

T  methods  are  followed  by  everyone,  and 
T  QA  activities  for  projects  are  planned  and  not  spontaneous. 

QA  must: 

T  measure  the  effectiveness  of  the  methods, 


▼  provide  visibility  to  management  via  appropriate  metrics  from 
prior  project  QA  experience,  and 

▼  know  when  the  methods  need  to  be  adjusted. 

▲  A  person  independent  of  the  political  and  organizational  chain- 
of-command  are  recommended  to  avoid  conflicts-of-interest 
to  achieve  appropriate  objectiveness  from  the  product  and 
its  stakeholders. 
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The  Technology 
Strategy  Companysm 


Techkoloqt  ’ 


QA  must  support  the  project  in  achieving  the 
intended  benefits  of  the  standards  the  project  sets 
for  itself. 


▲  If  the  project’s  processes  and  activities  do  not 
promote  or  support  its  standards,  policies,  or 
methods,  it’s  QA’s  job  to  bring  this  disconnect  to 
the  attention  of  the  people  who  can  make 
appropriate  changes. 
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QA  Distilled 


Technology  ™ 


▲  QA  boils  down  to  making  sure  that  the  things  that 
need  to  take  place  can  happen  and  are  happening, 
and  that  when  they  don’t  they  get  fixed. 

▲  Everything  else  is  technique. 

▲  “Keeping  things  simple”  is  critical  to  a  well-formed 
abstraction. 

▲  When  QA  is  distilled  to  the  above  statements, 
possibilities  are  created  regarding  how  to  look  at 
the  organization’s  QA  processes  so  that  they  can 
operate  in  any  environment. 
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Processes  within 
Lightweight  Environments 
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▲  Lightweight  development  doesn’t  mean  there  are 
no: 

▼  requirements  management, 

▼  QC, 

▼  QA, 

▼  CM, 

▼  project  planning, 

▼  project  tracking,  or 

▼  reviewing  of  designs  and  work. 

▲  Development  without  those  things  would  be  called 
stupid  programming,  not  agile  programming. 


Translatinq  techno  to  qy  dollars  into  business  senses 


Agenda 


▲  Introduction 

▲  Understanding  Agile 

▲  Role  and  Goal  of  “QA” 

▲  Historical  Role  of  QA 

▲  QA  in  Business 

▲  A  Word  About 
Development 
Processes 
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▲  QA  in  the  Context  of 
Development 
Processes 

▲  Rethinking  the  Quality 
Abstraction 

▲  An  Implementation 
Example  with  Scrum 

▲  Conclusion 
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Rethinking  the  Quality 
Abstraction 


The  Technology 
Strategy  Companysm 


▲  Challenges  of  Current  [typical]  QA  Approaches 

▲  Necessary  Value  and  Effectiveness  of  QA 

▲  Real  or  Perceived  “Pro-Active”  Effort 

▲  QA  Processes  Needed  by  the  Market  and 
Development 

A  Technology  for  Real-Time  Analysis 
A  Transforming  the  Abstraction 
A  Desired  “Target”  Abstraction 
A  Section  Summary 


Challenges  of  Current 
(typical)  QA  Approaches 


The  Technology 
Strategy  Company31* 


Technoldht  1 


Many  QA  processes  rely  on  generating 

▼  artifacts, 

▼  evidence,  and 

▼  other  labor-intensive  “bread  crumbs” 

▼  tangential  to  the  work  being  done  on  the  product 
itself. 

These  tangential  efforts  rely  on  the  same  people  as 
development  and  therefore  cannot  occur  on  top  of 
production,  therefore  increasing  the  amount  of  time 
it  takes  to  carry  out  a  project. 


▲  Stove-Piped 


^  ^ 
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Necessary  Value  and 
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▲  This  approach  to  the  QA  process: 

▼  relegates  QA  to  the  role  of  policing  and  gate-keeping, 

▼  drastically  minimizes  the  positive  impact  of  the 
overall  effectiveness  of  the  QA  program. 

▲  One  can  seriously  [and  not  without  merit]  question: 

▼  timeliness, 

▼  contribution,  and 

▼  overall  value 

▼  of  QA 

▲  ...  when  the  activities  defined  by  or  for  QA 
purposes  do  not  benefit  the  project. 
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Real  or  perceived 
“Pro- Active”  Effort 
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Techkoloqt  ’ 


▲  “Proactive”  QA  is  still  seldom  /Proactive  throughout 
the  lifecycle  of  development. 

▲  Proactive  often  means  a  level  of  effort  before  a 
project  starts,  followed  by  periodic  or  event-driven 
reactive  activities  that  are  only  conducted  as 
events  unfold. 


▲  It’s  this  entire  approach  that  needs  “re-thinking”. 
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QA  Processes  needed 
by  the  Market  and 
Development 

▲  NEED:  Process  that: 

▼  ensure  processes  are  matched  to  project  objectives  before  the 
project  gets  under  way 

T  get  into  the  detail  of  the  standards  and  methods  so  that  when 
the  standards  are  followed  they  automatically  generate  the 
necessary  "proof”  of  process  compliance. 

A  DON’T  NEED:  Processes  that 

T  simply  create  automated  markers  and  flags,  or 

▼  reinvent  the  "wheel” 

A  INSTEAD: 

▼  approaches  that  enmesh  metrics  and  data  generation  into  the 
development  process  so  that  the  successful  output  of  the 
process  is  only  possible  if  the  process  was  properly  followed. 
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▲  Use  the  technology  of  the  development  process, 
tools,  and  standards  as  the  medium  to  collect 
process  and  tracking  data 

▲  Instead  of  policing  the  processes  through  post¬ 
mortem  artifacts,  QA  could  be  free  to  analyze  the 
effectiveness  of  processes  in  real  time  and  make 
adjustments. 
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Transforming  the 
Abstraction 
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The  actual  abstraction  transformation  is  simple. 

Instead  of  focusing  the  quality  process  on  the  effort 
of  proving  a  formal  process  is  followed,  ensure  that 
the  processes  are: 

▼  effective, 

▼  productive,  and 

▼  valuable  to  the  goals  of  the  business,  and 

Create  production  methods  that  produce  the 
evidence  as  a  by-product  of  the  effort  rather  than  a 
separate  activity. 
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Abstraction 


Te-oh  nolost  ’ 


Transforming  FROM: 

▼  A  reactive,  investigative,  and  stove-piped  approach 

TO: 

▼  a  productive,  business-driven,  value-focused  umbrella 
of  activities  that  improve  the  development  effort 

Will  achieve  the  “rethinking”  of  the  QA  abstraction 
that  is  necessary  for  lightweight  development 
methods. 
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Section  Summary 
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Go  from  this: 


Tech  nolo qy  1 


Ordinary  implementation  of  QA  in  development 
environments. 


Development 

Processes 


Typical  QA 
Processes 


Stopping 

Points 


▲  QA  processes  are  in  super-imposed  onto 
development  processes. 

▲  Add  a  layer  of  effort  not  in-line  with  productivity 
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To  this: 

▲  Preferred  implementation  of  QA  in  development 


Development 


Processes 


Processes 


Points 


▲  QA  processes  are  integrated  into  and  aligned  with 
development,  increasing  development  productivity. 

▲  Contributes  to  capacity  and  value  of  company. 
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Agenda 


▲  Introduction 

▲  Understanding  Agile 

▲  Role  and  Goal  of  “QA 

▲  Historical  Role  of  QA 

▲  QA  in  Business 

▲  A  Word  About 
Development 
Processes 
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Teghmolc 

▲  QA  in  the  Context  of 
Development 
Processes 

▲  Rethinking  the  Quality 
Abstraction 

▲  An  Implementation 
Example  with  Scrum 

▲  Conclusion 
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An  Implementation 
Example 


Collect  the  desired  practices  that  add  the 
desired  discipline. 

Find  the  actual  work  being  done  at  a  given 
location. 


The  Technology 
Strategy  Companysm 


Techno  loot  ’ 


Insert  the  practices  into  where  the  work  is 
done. 


ID/Define  life  cycles  in  which  actual  work 
happens. 

Centralize  redundant  policies,  processes, 
procedures  and  templates. 
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...HOW  MUCH  ARE 


Tech nolo qy  1 


Policy 


Establishes  that  company  projects  will  adhere  to  formal 
processes  and  states  company’s  policy  for  quality  values,  quality 
work,  and  how  these  align  with  the  company’s  mission  and  vision. 


Quality 

Manual 


Outlines  what  company  does  to  ensure  on-time,  on- 
budget,  fully  featured/functional  projects. 


Expectation/ 


Corp  /BizDev 


Fulfillment 


Life  Cy 


Outlines  the  phases  of  every  project  @  company  and  scopes 
activities  and  deliverables  within  each  phase. 

Establishes  each  projects  parameters. 


Engagement/ 


Mgmt/Tech 


Life  Cycle 
[menu] 


A  menu  of  management  or  technical  activities  that 
each  project  can  choose  from  as  appropriate.  Each 
project  is  required  to  identify  a  life  cycle.  This  menu 
provides  the  list  of  what  can  be  in  a  life  cycle. 


Daily/Weekly 

Management 


Specifies  how  projects  are  managed. 


w 


Translatinq  technoloqy  dollars  into  business  sense r 


©2005  Entinex,  Inc.  All  rights  reserved.  PROPRIETARY 


23  November  2005 


69 


The  Technology 
Strategy  Company31* 


...  AND  THEN  SOME... 


Policy 


Quality 

Manual 


Specific 

y><yM 

Company 

Fulfillment 

Process 

4/\ 

Area 

Specific 

rv 

Life  Cvcle 

Process 

Policies 

Process 

Area 

Process 

Mgmt/Tech 
Life  Cycle 

H'V  4 

151 

Ta  * 

n 

Descriptions 

[menu] 

Daily 
IManagementl 
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What’s  in  the  Quality 
Manual? 


The  Technology 
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Explains  how  on  each  project, 

all  company  Processes: 

▼  are  planned-out  and  tailored 
from  a  single  set  of  company 
processes 

▼  are  assigned  as  someone’s 
responsibility 

▼  are  provided  resources  to  be 
done 

▼  are  assured  of  having  people 
trained  in  them 

▼  have  their  work  products 
configuration  controlled 

▼  involve  relevant  stakeholders 

▼  are  monitored  &  controlled 

▼  are  objectively  evaluated 
against  applicable  standards, 

▼  have  performance  reviewed 
with  higher  management,  and 

▼  incorporate  lessons  learned 
for  improvement 
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lO 


Quality 

Manual 


Work-Product 

Generation 


Company 
Fulfillment 
Life  Cvcle_^^^| 

^^^Mgmt/Tech 
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Phase  1: 
Initial  Analysis 
&  Response 


Phase  2: 

Planning/Kick-Off 


Phase  3: 
Follow-Through 


Phase  4: 
Close-Out 


23  November  2005 


74 


Translating  technology  dollars  into  business  sense.SM 


©2005  Entinex,  Inc.  All  rights  reserved.  PROPRIETARY 


▲  Get  from  RFP  to  Award 


and/or  from  Award  to  Start 


▲  Provides  a  business  basis  for 
going  forward 

▲  Provides  requirements  against 
which  to  manage  the  initial 
activities 


▲  Scopes  the  project  before 
details  are  known 


▲  Breaks  out  of  the  Catch-22  of 
“when  does  the  project 
start?” 


▲  Allows  for  minimal  mock-ups 
or  prototyping/engineering 
analysis  to  obtain  project 


requirements  agreement. 
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K.r- 


NlfBS 


▲  Identifies  the  project’s: 

▼  Type 

▼  Management  or  Technical 
Life  Cycle 

▼  Major  Product  and  Document 
Deliverables 

▼  Major  Tasks 

▼  Assignments,  Roles  and 
Stakeholders 


▼  Resources,  Tools  and  Assets 

▼  Plans 

▼  Project  Monitoring  Events 

▼  Milestones 

▼  Required  Training 

▼  Measures  &  Analyses 
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Phase  3  Concepts 


Phase  1: 
Initial  Analysis 
&  Response 


Phase  2: 
Planning/Kick-Off 


Phase  3: 
Follow-Through 


Phase  4: 
Close-Out 
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All  detailed  engineering  and 
provisioning  of  the  solutions 
and  products 

Execution  of  the  entire 
Management  or  Technical 
Life  Cycle 

From  Design  through  Delivery 
and  Installation 

Can  be  iterative  with  Phase  2 

All  phases  of  the  daily 
process  through  Closure 
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Phase  4  Concepts 


Phase  1: 
Initial  Analysis 
&  Response 


Phase  2: 
Planning/Kick-Off 


Phase  3: 
Follow-Through 


Phase  4: 
Close-Out 


Translating  technology  dollars  into  business  sense.SM 


©2005  Entinex,  Inc.  All  rights  reserved.  PROPRIETARY 


The  Technology 
Strategy  Companysm 


Opportunity  for  Lessons 
Learned 

Final  Administrative  Checks 

Customer  Feedback 

Final  PPQA  Checks  & 
Audits 

Final  CM  Audits 


1 
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Checkpoint  reviews  check 
phase  templates  and  other 
work  products  to  ensure 
phase  tasks  are  completed 
before  getting  too  far  and 
deep  into  the  next  phase. 

The  final  review  is  a  “close¬ 
out”  to  collected 
performance  data. 


Translatinq  technoloqv  dollars  into  business  sense.SM 


©2005  Entinex,  Inc.  All  rights  reserved.  PROPRIETARY 


The  Technology 
Strategy  Companysm 


heckpoi 


Phase  4: 
Close-Out 
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PPQA  Concepts  in 
Work  Products 
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Standards  Verification 
performs  process  checks 
against  company’s  own 
standards 

Engineering  Reviews 
perform  integrity  checks  on 
designs,  analyses,  and 
solutions 

Peer  Reviews  &  Testing 
perform  product  checks  on 
code  and  code-based  work 


Templates 


Mgmt/Tech 
Life  Cycle 


Backlogs  & 
Peer 
Reviews 


Techkoloqt  ’ 


Pb± 


W\ 
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All  Other 


Technolost ’ 


▲  All  other  practices  within  process  areas  have  been 
distributed  into  and  made  seamless  with  company 
planning  and  engineering  activities. 

▲  Some  practices  are  performed  once  and  passed 
through  with  each  project  review. 

▲  Some  practices  are  addressed  by  merely  including 
an  item  on  a  meeting  agenda. 
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Agile  CMMI 


Business 

riNESsf—  /  ’ 

I  # 


fi 


s 


CMMI 


TM 


CMP  gb  mife  dr  iks  Mm  fimgffMis 
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▲  QA’s  real  job  is  to  ensure  certain  information  is 
generated  to  assure  that  processes  are  followed. 

▲  There  may  be  no  need  for  QA  to  get  mired  in  the 
specifics  of  development. 

▲  By  collaborating  with  developers  on  producing  what 
QA  needs, 

▼  the  every-day  hour  to  hour  activities  of  development 
can  become  part  of  development  activities  and 

▼  QA  can  be  left  to  monitor  the  overall  effectiveness 
of  the  project’s  processes  and  feed  back  process 
improvements. 
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QA  IN  A 

Project 
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A  In  a  well  integrated  project: 

▼  generating  the  data  QA  needs  would  merely  be  a 
report  that  runs  every  so  often  querying  certain 
tables  and  build  repositories. 

▲  A  QA  program  at  this  level  of  abstraction  is: 

▼  infinitely  scalable  to  any  project 

▼  as  long  as  there’s  the  will  to  cooperate 

▼  for  the  purposes  of  benefiting  the  business. 
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CMMI  with  Scrum 

▲  Product  Backlog  and  Planning 

▲  Sprint  Backlog  and  Planning 

▲  Resource  Allocation 

▲  WBS 

▲  Daily  Team  Meetings 

▲  Peer  Reviews  and  Inspection 

▲  Sprint  Review 
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Product  Backlog  and 
Planning 
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▲  The  product  backlog  is 

▲  REQM 

defined  by  the  product 

owner  and  managed  by  the 

▲  PP 

Scrum  master. 

▲  PMC 

▲  Defines  High  Level 

Requirements  and  sets 

priorities. 

▲  CM 

▲  Defines  high  level  work 

▲  GP  2.2,  2.3,  2.4,  2.7 

break  down  structure. 

▲  [RD,  TS,  PI,  IPM,  RISK, 

▲  May  define  high  level 

DAR] 

release  schedule. 

Translating  technology  dollars  into  business  sense.SM 

Sprint  Backlog  and 
Planning 


▲  Breaks  the  product  goals 
down  into  demonstrable  goals. 
This  is  usually  at  the  use  case 
level. 

A  Tasks  are  broken  down  into 
hour-based  estimates, 
anything  over  1 6  hours  was 
broken  down  into  smaller 
pieces. 

A  The  team  creates  tasks, 
estimates  and  determines  who 
is  going  to  do  what,  everyone 
commits  to  the  feasibility  of 
the  plan. 

▼  What  can  be  done  in  30  days 
with  the  resources  we  have 
at  our  disposal? 
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▲  REQM 

▲  PP 

▲  PMC 

▲  CM 

▲  GP  2.2,  2.3,  2.4,  2.6,  2.7 

▲  [RD,  TS,  PI,  VAL,  VER, 

IPM,  RISK,  DAR] 
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Managed  by  the  team,  as 
members  commit  to  getting 
the  work  done. 

Members  can  play  many 
roles  at  the  same  time: 

▼  Developer,  Architect  and 
DBA 

▼  Developer,  Tester  and 
Requirements  Analyst 

Member  are  committed  to  the 
project  and  external  noise  is 
minimized. 

The  Scrum  Master  helps 
alleviate  resource  contention 
and  noise. 
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REQM 


PP 


PMC 


MA 


CM 


GP  2.2,  2.3,  2.4,  2.7 


[RD,  TS,  PI,  IPM,  RISK,  DAR] 


Work  Breakdown 
Structure 


a  A  Product  Goal  can  be  broken 
down  into  many  use  cases 

▼  "The  application  needs  to 
contain  a  shopping  cart” 

A  A  Sprint  Goal  satisfies  a  use 
case 

▼  “Allow  a  registered  use  to 
put  items  into  their  shopping 
cart” 

▼  “Allow  a  user  to  update  the 
quantities  in  the  shopping 
cart” 

A  Each  sprint  goal  is 

demonstrabe,  releasable 
functionality. 

▼  Show  that  this  use  case 
works,  and  has  been  tested 
and  could  be  released  as 
functionality 


Translatinq  techno  to  qv  dollars  into  business  sense.SM 


©2005  Entinex,  Inc.  All  rights  reserved.  PROPRIETARY 


The  Technology 
Strategy  Companysm 


▲  REQM 

▲  PP 

▲  PMC 

▲  CM 

▲  GP  2.2,  2.3,  2.4,  2.7 

▲  [RD,  TS,  PI,  IPM,  RISK, 
DAR] 


23  November  2005 


89 


The  Technology 
Strategy  Company31* 


Daily  Team  Meetings 


Technology  1 


Quick  1 5-30  Minute  Stand  up 
Meetings. 

Answer  3  Questions: 

▼  What  have  you  done  since  the 
last  meeting  ? 

▼  What  are  you  going  to  do  before 
our  next  meeting  ? 

▼  What  issues  are  you  having  that 
are  impeding  progress  ? 

Daily  Inspection  and  Visibility  into 
team  progress. 

Daily  Issues  Management  and 
Resolution. 

Daily  Project  Command  and 
Control  within  the  self  managing 
team. 


REQM 


PP 


PMC 


MA 


PPQA 


CM 


GP  2.2,  2.3,  2.4,  2.6,  2.7,  2.8, 
2.9,  2.10 

[RD,  TS,  PI,  IPM,  RISK,  DAR] 
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peer  review  and 
Inspection 

▲  Peer  reviews  keeps  the  team 
members  honest. 

A  Peer  reviews  are  about 

mentoring,  not  policing. 

A  Complete  checkpoints  and 
tollgates  along  the  project 
road  map  that  can  be  done 
iteratively  and  kept  non- 
invasive. 
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REQM 


PMC 


MA 


PPQA 


CM 


GP  2.6,  2.7,  2.9,  2.10,  3.2 


[RD,  TS,  PI,  VAL,  VER,  IPM, 
RISK,  DAR] 
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Sprint  Review 


▲  The  Sprint  review  is  a  form  of 
validity  check-it  is  determined 
that  the  right  product  is  being 
built. 

A  Covers  whether  the  product 
was  built  right  because  a 
working  version  of  the  product 
is  giving  a  viewing  to  the 
product  owner. 

A  Product  Owner  [s]  decides  if 
functionally  and  quality  are 
sufficient  to  be  released 
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heckpoi 


Phase  3: 
Follow-Through 


heckpoi 


Phase  4: 
Close-Out 
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Agenda 


▲  Introduction 

▲  Understanding  Agile 

▲  Role  and  Goal  of  “QA” 

▲  Historical  Role  of  QA 

▲  QA  in  Business 

▲  A  Word  About 
Development 
Processes 
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▲  QA  in  the  Context  of 
Development 
Processes 

▲  Rethinking  the  Quality 
Abstraction 

▲  An  Implementation 
Example  with  Scrum 

▲  Conclusion 
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Conclusion 


Reversing  Common  Misconceptions  About  Agile 


▲  Re-Cap 


References 


Q&A 


TECHNOLOGY  ™ 
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Reversing  Common 
Misconceptions 
About  Agile 
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Technolost ’ 


▲  Lightweight/Agile  is  not  about  eliminating 
processes. 

▲  Lightweight/ Agile  is  about  sufficient 
processes  to  achieve  the  project’s 
objectives,  but  only  those  processes  that 
are  necessary. 

▲  QA  and  other  development  processes 
have  not  kept  up  with  the  changes  in 
development  methods  or  technologies. 

▲  Agile  processes  have  everything  the  agile 
project  needs  to  get  accomplished. 
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Re-cap 

▲  History  of  QA  Abstractions 

▲  Context  of  the  Role  of  QA 

▲  Mutual  Distaste  Between  Lightweight  Developers 
and  Process  Discipline  Practitioners 

▲  Create  processes  that  Promote  Productivity 

▲  Middle  Ground  Where  Diligently  Followed 
Lightweight  Methods  Find  Mutually  Beneficial 
Ground  with  Process  Discipline 

▲  Result:  Excellent  Software  of  Very  Good  Value. 
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▲  Today’s  s/w  market  is  not  that  of  1 5  +  years  ago. 

A  Traditional  QA  fails  on  s/w  because  the 
manufacturing  model  for  development  is  a  failure. 

A  QA  must  be  integrally  aligned  with  development 
processes  and  business  goals  to  survive  in  today’s 
market. 

A  Agile  development  processes  will  survive  because 
they’re  more  consistent  w/SW  realities. 

A  Agile  can  be  disciplined  if  discipline  can  be  agile. 
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Technology  ™ 


▲  Beck,  Kent.  Extreme  Programming  Explained:  Embrace  Change ,  Addison- 
Wesley,  2001 . 

▲  Glazer,  Hillel.  “Dispelling  the  Process  Myth,”  CrossTalk,  November  2001 ,  Vol. 
14  No.11,  pp.  27-30. 

▲  http://alistair.cockburn.us 

▲  http  ://c2 .  com/cgi/wiki?XpAndTheCmm 

▲  http  ://c2 .  com/cgi/wiki?CategoryPattern 

▲  http  ://c2 .  com/cgi/wiki?QalsNotQc 

▲  http://www.extremeprogramming.org 

▲  http://www.martinfowler.com/articles/newMethodology.html 

▲  http://www.sei.cmu.edu 


CMM®  and  CMMI®  are  Registered  Trademarks  of  the  Software  Engineering 
Institute. 


A%/e  CMMI®  is  a  Trademark  of  Entinex,  Inc. 
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Common  Systems  Engineering  Management  &  Technical  Issues 


Critical  Program  Performance  Challenges.... 

•  Obtaining  a  realistic  understanding  and  managing  internal  and  external 
customer  requirements. 

•  Lacking  verified  and  validated  techniques  of  measuring,  controlling  and 
balancing  cost  and  performance  requirements. 

•  Hiring  the  “right  staff”  in  time  to  evaluate  and  implement  emerging 
technologies. 

•  Maintaining  ever-increasing  program  profitability  goals  due  to  the  impact  of 
emerging  administration  and  technical  issues,  risk,  and  changing  customer 
environments. 

•  Sustaining  multi-year  technical  service  and  product  support  levels  is  impacted 
by  increases  in  costs,  staff  transitions  and  changing  customer  requirements. 
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Background  of  Journey 


Rationale  for  Initiating  Journey 

Faced  with  extreme  challenges  of  maintaining  profitability  while  managing  increasing 
performance  costs  and  concurrently  responding  to  a  dynamically  changing  customer 
environment 


Organization  Overview 

Organization  supported  customer  by  performing  on-site  and  off-site  engineering 
and  scientific  services  and  product  development  for  a  wide  assortment  of 
space  based  platforms. 

Kick-Off  Activity 

Multi-domain  leadership  team  assembled  to  plan  the  multi-year  journey  to 
higher  maturity  levels.  The  initial  version  of  the  plan  launched  pilot  projects 
in  the  small  software  development  organization  followed  by  support  functions  of 
finance,  procurement  and  HR. 

Obstacles 

The  organization  faced  initial  obstacles  of  resources  to  construct  a  framework  to  integrate 
key  program  and  technical  functions  as  well  as  staff  training  in  the  CMMI®. 
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Engineering  Approach  to  Developing  a  Program  Performance  Model 


The  leadership  team,  composed  of  engineers,  developers  and  scientists, 
constructed  the  framework  for  the  program  performance  model  using  SE 
Vee  life  cycle  model. 


Application  of  the  practices  in  the  CMMI®  Process  Areas  (PAs)  were  used 
across  the  program  and  projects  to  implement  the  relevant  phases  in  the 
SE  Vee  model. 


Program  Performance  Model 

•  Functioned  as  a  risk  management  tool 

•  Balanced  cash  flow,  staff  size,  product  quality  and  customer 
satisfaction 

4 

•  Sustained  service  levels  and  technical  performance  at  planned  costs 


Challenges  in  Developing  the  Program  Performance  Model 


Time  Factors 


Realistic  understanding  of  continually  evolving  customer 
environments 


Exponential  increase  in  costs 
downstream 


Developing  and  implementing  validated  techniques 
to  balance  cost  and  performance 

Availability  of  global  rapidly  emerging  technologies 


Mismatch  in  technical  performance 
requirements  versus  program  budget 

Inflexible,  non-scalable  designs 


Impact  of  operational  changes 


System  requirements  obsolete 


Life  cycle  planning 


O&M  infrastructure 
costs  vs.  service  levels 


Staffing 


Unfilled  positions  lower 
revenue 


About  the  SE  Vee  Model 


Operations 
Concept 

Architecture 

Design 

Development 


Operations 
(including 
D  &  D) 


Deployment 


Verification 


Integration 


Test 


Maturit 


■  The  SE  Vee  Life  Cycle  Model  presented  to  the  Texas  Board  of  Professional  Engineers,  1999, 
by  Arunski,  Martin,  Brown  and  Buede. 

■  The  phases  in  the  Vee  are  traditionally  applied  to  engineering  products  and  services  such  as 
weapons  systems,  communications  networks  and  technical  support. 

■  In  any  program,  phases  in  the  Vee  may  not  be  performed  or  applicable  or  may  exist 
in  numerous  projects  at  different  times. 


Key  infrastructure  functions,  such  as  finance,  contracts,  and  HR  benefit 

from  implementing  the  same  engineering  discipline  and  activities  as  technical  projects. 
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Engineering  of  Program  Performance  Models 


Operations 


Operations 


Concept 

m 

Architecture 
Design 
Development 


“Vee”  Activity 

Operation 

Concept 

Architecture 


Design 


Example  Critical  Support  Functions 

Resources  (space, 
accounting,  BP  systems) 

Business  goals  performance  intervals 

Structure  of  business  performance 
interfaces  (receivables,  quality  measures 
inventory,  growth) 

Performance  constraints  for  cash  flow, 
service  level  performance,  staff  size 


Development 


Increments  to  support  planned  site  expansion 
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Engineering  of  Program  Process  Performance  Models 

CMMI  Process  Area 
Categories 

Project  Management 

(Project  Planning,  Project  Monitoring  &  Control, 

Risk  Management,  Integrated  Project  Management, 
Integrated  Teaming,  Integrated  Supplier  Management, 
Quantitative  Project  Management) 


Concept 

Architecture 

Design  M 

Development 

Verification 

Integration 

Test 


Process  Management 

(Organizational  Process  Focus, 

Organizational  Process  Definition, 
Organizational  Training,  Organizational  Process 
Performance,  Organizational  Innovation  and 
Deployment) 


Engineering 

(Requirements  Management, 

Requirements  Development,  Technical  Solution, 
Product  Integration,  Verification,  Validation) 


Operations 

Deployment 


Support 

(Configuration  Management, 

Process  &  Product  Quality  Assurance, 

Measurement  &  Analysis,  Causal  Analysis  &  Resolution, 
Decision  Analysis  &  Resolution, 

Organizational  Environment  for  Integration) 


Engineering  of  Support  Function  Framework 


Operations 
Concept 

Architecture 


Design 

Development 


If 


Operations 

Deployment 


Verification 

Integration 


Test 


CMMI  Product 
Suite 


“Vee”  Phase  Example  Key  Support  Functions 


Key  CMMI  PAs 


Operations 

Concept 

Architecture 

Design 

Development 


Resources  (space,  BP  systems,  staffing  levels) 

Business  goals  performance  intervals 

Structure  of  business  performance 
interfaces  (cash  flow,  quality  measures, 
inventory,  growth,  .etc.) 

Performance  constraints  for  cash  flow, 
service  performance,  staffing 


M&A,  PP,  RSKM 
M&A,  RD 
M&A,  TS,  PI 


M&A,  RD,  RM,  TS 


Builds  to  support  planned  market  and  program  M&A,  RD,  PP,  RSKM 
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Engineering  of  Support  Function  Framework  (Continued) 


Operations 

Concept 


Architecture 
Design 
Development 


Operations 

Deployment 


Verification 

Integration 


Test 


“Vee”  Phase 

Test 

Integration 

Verification 

Deployment 

Operations 


Examples  Key  Support  Functions 

Finance  test  scenarios  and  databases 

New  interfaces  of  components  (acquisitions) 
for  growth  goals,  finance  and  HR  functions 

Invoicing  and  staffing  processes 

Perfective  and  adaptive  maintenance  of  support 
functions 

Forecasting  of  staffing  and  facilities  costs 


Key  CMMI  PAs 

M&A,  VER,  VAL 
TS,  PI 

M&A,  VER,  VAL 
PP,  PMC,  TS 

PP,  PMC,  QPM,  10 
OPP,  OID 


Overview  of  the  SE  Vee,  CMMI  Process  Areas  and  Business  Goals 


Operations 


Operations 


Concept 

Architecture 

Design 

Development 


Vee 


Deployment 


m  • 


Verification 

Integration 


Test 


Key  Process  Areas 

Measurement  &  Analysis,  Project  Planning,  Project  Monitoring  &  Control, 

Risk  Management,  Quantitative  Project  Management,  Organizational  Innovation  &  Deployment, 
Causal  Analysis  &  Resolution,  Decision  Analysis  &  Resolution... 


. . 


Program 
Business 
Goals  / 


Support 


Technical 


Customer 

Satisfaction 


Invoicing  Latent  Defects  Customer 

Procurement  Satisfaction 

Contracts 
Staffing 
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Example  of  Balancing  Cost  and  Technical  Performance 

in  a  Small  Setting 


Key  CMMI  Process  Areas  j 

Measurement  &  Analysis,  Project  Planning,  Project  Monitoring  &  Control,  | 

Risk  Management,  Quantitative  Project  Management,  Organizational  Innovation  &  Deployment; 
Causal  Analysis  &  Resolution,  Decision  Analysis  &  Resolution...  i 


Practices 


Program 

Business 

Goals 


Support 


Technical 


Customer 

Satisfaction 


Invoicing 

Procurement 

Contracts 

Staffing 


Latent  Defects 


Customer 

Satisfaction 


Key 

Measurements 


Process 

Performance 

Intervals 


Catches/Escapes 


Customer 

Satisfaction 
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Case  Study  Example  of  Balancing  Cost  and  Technical  Performance 

in  a  Small  Setting  (Continued) 


Program 

Business 

Goals 

Key 

Measurements 


Support 


Invoicing 

Procurement 

Contracts 

Staffing 


Technical 


Customer 

Satisfaction 


Latent  Defects  Customer 

Satisfaction 


Process 

Performance 

Intervals 


-6.0%  <  Staff  Size  Accuracy  >  9.0%  o  <  Latent  Defects  >  3  4.5  <  Customer  Satisfaction  >  5.0 

-8.1%  <  Invoice  Accuracy  >  6.5% 
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Lessons  Learned  During  the  Journey 


■  Focus  on  defining  business  goals  and  related  measurements  for  the 
organization  for  the  entire  period  of  program  performance. 

■  Plan  and  implement  the  applicable  CMMI  PA  practices  in  projects 
across  the 

organization  sooner  rather  than  later  as  retrofitting  is  difficult. 

■  Measurement  processes  should  focus  on  forecasting  yearly  costs, 
required  technical  performance  levels,  quality  goals  and  program 
support  levels. 

■  Apply  SE  tools  and  techniques,  such  as  alternative  evaluations, 
performance  simulations,  requirements  definition  and  risk  analysis 
across  the  infrastructure  functions  as  well  as  technical  services  using 
practices  in  the  CMMI. 

■  Provide  CMMI  training  to  classes  with  diverse  backgrounds  to  enhance  14 
team  building. 


Lessons  Learned  (Continued) 


The  phases  in  the  SE  Vee  provide  a  useful  and  applicable 
life  cycle  model  for  engineering  of  a  framework  to  integrate 
management  and  technical  practices  across  a  program. 


The  SE  Vee  is  very  adaptable  to  small  settings  and  applies  to  support  services, 
such  as  finance,  contracts  and  HR. 


The  practices  in  the  current  version  of  CMMI  Process  Areas  cover  a  large 
percentage  of  the  phases  in  the  Vee. 

Customer  advocacy  and  participation  in  an  appraisal  is  very  advantageous  for  all. 

For  best  results,  focus  on  first  defining  business  goals  and  relevant 
measurements  to  implement  continuous  process  improvement  to  achieve 
a  program  performance  model  to  balance  cost  and  technical  performance 
via  the  CMMI. 


Expect  multi  iterations  during  the  measurement  and  analysis  activities  before 
key  sub-processes  are  controlled.  15 
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Background 

■  Appraisal  results  show  some  common  weaknesses  for  Level  4 
and  5 

■Tracing  back.... 

-Time  pressures  to  get  the  level 

-Wrong  decisions  at  key  points 

-  Relationship  to  current  processes 
ignored 

-  Statistics  takes  precedence  over 
good  business  decisions 

■  How  to  avoid  missteps 

-Have  a  guide 

-  Integrate  new  activities  with  current 
activities 

-  Interpret  for  your  environment 
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Agenda 


■  Commonly  Cited  Level  4  and  5  Problems 

■  Key  Decision  Points  Along  the  Way 

-  How  Level  4  and  5  processes  are  developed 
-Compose  the  Define  Process 

-Selecting  Subprocesses  for  Statistical  Management 
-Choice  of  Statistical  Techniques 
-Statistical  and  Quantitative  Management 
-What  Characterizes  Level  4  Institutionalization? 

-  Using  Six  Sigma  for  Maturity  Level  5 
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Commonly  Cited  Level  4  and  5  Problems 


■  Business  goals  not  aligned  with  measures 

■  Failure  to  revise  measurements  (or  question  validity) 

■  Relationship  between  statistically  managed  subprocesses  and 
business  goals  is  unclear 

■  Failure  to  perform  risk  mitigation  when  desired  results  do  not 
match  expected  results 

■  Models  aren’t  used  to  manage  attainment  of  critical  project 
objectives 

■  Statistical  techniques  are  used  incorrectly 

■  Failure  to  question  and  or  evolve  measurements 

■  Level  4  and  5  activities  are  unrelated  (including  Six  Sigma 
activities) 
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How  Level  4  and  5  Processes  are  Developed 


□  We  develop  new  processes,  add  them  to  our 
Process  Asset  Library,  and  transition  projects  as 
needed 


We  evolve  existing  processes  to  include  level  4  and 
5  activities  where  appropriate 


■  Level  4  and  5  activities  do  not  replace  existing 
processes 

■  Level  4  and  5  activities  are  extensions  of  existing 
Process  Areas 

-Measurement 
-Project  Management 
-Process  Improvement 
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How  Processes  Evolve 


Compose  the  Defined  Process 
How  do  you  compose  the  defined  process? 

[JTWe  use  our  project  objectives  to  determine  our  defined  process 

□  We  follow  the  tailoring  guidelines  to  determine  our  defined 
process 

■  If  project  objectives  (desired)  are  not  achievable  with 
historical  achievements  (expected) 

-Current  tailoring  won’t  achieve  different  results 

-Risk  needs  to  be  identified  and  analyzed  (CAR  or  Six 
Sigma  could  help) 

-What  needs  to  be  added  or  changed  to  achieve  project’s 
objectives? 

■  Defined  process  expectations  may  not  be  known 

-Can  model  be  used  to  monitor  risk? 

-How  will  you  gain  insight  into  the  impact  of  different 
processes? 
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Selecting  Subprocesses  for  Statistical  Management 


We  select  subprocesses  that  are  critical  to  meeting 
our  project  objectives 


□  The  subprocesses  we  select  are  consistent  across 
the  organization 


■  Project  needs  and  organizational  needs  may  be 
different  -  contract  type,  customer,  product  needs 

■  Combining  data  across  projects  to  increase 
confidence  is  problematic 

-Variation  is  usually  increased 

-Valuable  insight  into  needed  process 
performance  can  be  lost 
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Choice  of  Statistical  Techniques 


We  rely  primarily  Statistical  Process  Control  (SPC) 
techniques 

We  encourage  a  wide  range  of  statistical  techniques 


■  SPC  techniques  work  well  for  some  situations 

-Data  should  be  time  independent 

-Sufficient  data  exists  for  confidence 

-Calculated  control  limits  are  useful 

-Collect  enough  information  so  that 
data  can  be  repartitioned  if  needed 

■  SPC  can  be  use  to  verify  results  of  other 
techniques 

-Design  modeling  and  simulation  with  manufacturing  SPC 
-Part-time  resource  allocation  and  productivity 


o  o  -  UCL 

O  O 

o - 5 - ° - Average 

°  °  °  *  -  LCL 
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Statistical  and  Quantitative  Management 


□  Statistical  characterization  indicates  statistical 
management 


Statistical  management  infers  acceptance  of 
statistical  expectations 


■  If  expected  results  will  not  satisfy 
desired  results  -  quantitative 
management  makes  sense 


USL 

LSL 

O  O  0  "  UCL 

u  o  o  °  Average 

°  °  -  LCL 

■  Statistical  management  may  not  be  good 
business  in  all  cases 


-Expected  variation  is  unacceptable 

-Data  is  insufficient  to  provide  sufficient  confidence 
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Organizational  Role  in  Quantitative  Management 


The  project  determines  what  will  be  managed  using 
statistical  and  other  quantitative  techniques 


The  organization  sets  guidelines  for  projects  as  to 
what  will  be  managed  using  statistical  and  other 
quantitative  techniques 


■  Organizational  role 

-Need  to  monitor  certain  indicators  at  organizational  level 

-Provides  historical  project  data  as  a  planning  asset  to 
projects 

■  Project  role 

-Satisfy  customer  needs  and  expectations 

-Organizational  obligation  for  insight 
into  future  projects  (CAR,  OID) 
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What  Characterizes  Level  4  Institutionalization? 

□  We  have  demonstrated  use  of  statistical  and  other 
quantitative  techniques  across  the  entire  lifecycle 

Sf  We  have  collected  enough  data  and  used 

techniques  long  enough  to  determine  if  it  is  working 

□  It’s  time  for  the  appraisal 

■  Is  it  working? 

-Projects  are  able  to  predict  and  insight  is  valuable 

-Unexpected  failures  are  analyzed  - 
revision  to  measurements  or  techniques 

-Stakeholder  involvement  and  confidence 
is  apparent 

■  It  makes  good  business  sense 

-Intent  of  model  is  satisfied 
-Business  and  Quantitative  objectives  are  integrated 
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Using  Six  Sigma  for  Maturity  Level  5 


We’ve  used  Six  Sigma  long  before  we  introduced  level 
4  activities 


□  Six  Sigma  projects  satisfy  Maturity  Level  5  activities 


A  subset  of  our  Six  Sigma  projects  satisfy  Maturity 
Level  5  activities 


■  Six  Sigma  has  numerous  interpretations 

-Some  rely  on  statistical  understanding 
-Some  require  use  of  statistical  techniques 

■  Look  for  Six  Sigma  projects  that  support  Maturity 
Level  4  activities 

■  Include  cost/benefit  estimations  and  tracking  to 
achievement  of  organizational/project  business 
objectives 
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Summary 


■  Understand  the  differences  between  Level  4  and 
Level  3  behaviors 

■Understand  the  relationship  and  evolution  of  Level  3 
to  Level  4  activities 

-Project  Management 
-Process  Improvement 
-Measurement 

■Interpret  the  activities  in  the  context  of  your  business 

-Level  4  and  5  activities  need  to  make  good  business  sense 
-Understand  the  big  picture  of  CMMI  Level  4  and  5 
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THE 

CENTER  FOR 

SYSTEMS  MANAGEMENT 

INTEGRATING  PROCESS  &  PERFORMANCE 


Q&A 


Kathy  King 

The  Center  for  Systems  Management 
Office:  (703)  852-3329 
Cell:  (703)623-7559 

kkinq@csm.com 

1951  Kidwell  Dr,  Suite  750 
Vienna,  VA  22182 
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Carnegie  Mel  Inn 

Software  Engineering  Institute 


CMMI 


Agenda 

What’s  Happening  in  the  International  Standards 
Arena? 

Capitalizing  on  Synergies  with  Selected  International 
Standards 

Current  Status 
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Carnegie  Mel  Ion 

Software  Engineering  Institute 


CMMI 


What’s  Happening  in  ISO? 

Work  wrapping  up  on  process  assessment  model  for  12207  - 
Software  Life  Cycle  Processes  (15504-5)  -  to  be  published  Q106, 

Work  beginning  on  process  assessment  model  for  15288  -  System 
Life  Cycles  Processes  (15504-6)  -  to  be  published  Q308  (estimate) 

Work  beginning  to  add  organizational  maturity  construct  to  15504 
(will  be  published  as  15504-7)  -  to  be  published  Q308  (estimate) 

Revision  of  15939  -  Software  Measurement  Process  -  to  encompass 
systems  and  software, 

Work  towards  eventual  “harmonization”  of  12207  and  15288  -  this 
COULD  result  in  a  single  IS  addressing  both, 

Work  beginning  on  applying  12207  to  Very  Small  Enterprises 

Work  beginning  on  Certification  of  Software  Engineers 
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What  Does  This  Mean!? 

National  boundaries  (and  all  which  that  implies) 
continue  to  become  less  relevant  to  international 
commerce, 

More  pressure  to  either  use  international  standards 
directly  or  ensure  that  “local”  standards  have 
meaningful  connectivity  to  relevant  international 
standards. 

More  pressure  for  “reciprocity”  agreement 
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Carnegie  Mel  Ion 

Software  Engineering  Institute 


Impact  of  Globalization  Pressures  -1 


SEI  involvement  in  relevant  international  standard  work 
since  early  90s, 

CMMI  Product  Suite  manifestations: 

•  A-spec  for  CMMI  Product  Suite  cites  as  reference 
documents  “Applicable  ISO/IEC  documents,  including 
ISO/IEC  12207  and  ISO/IEC  15504”;  contains  two  “line 
item”  requirements  citing  15504, 

•  ARC  draws  heavily  on  15504-2  requirements, 

•  SCAMPI  (A)  is  largely  15504  compliant, 

•  Nine  standards  (sector,  professional  society  or 
international)  are  cited  in  the  CMMI  model  reference 
appendix, 

•  Six  standards  (sector,  professional  society  or 
international)  are  cited  in  the  CMMI  model  glossary 
“order  of  precedence” 
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Carnqrie  Mel  Ion 

Software  Engineering  Institute  CMMI 

Impact  of  Globalization  Pressures  -2 

(2000)  Publication  of  Spice  for  Space  method  and  model  (S4S)  by 
ESA  for  use  by  the  European  space  industry, 

(2001)  Publication  of  Spice-9000  for  Space  method  and  model 
(S9kS), 

(2003)  Publication  of  mappings  relating  CMMI  to  9001  by  Boris 
Mutafelija  and  Harvey  Stromberg, 

(2004)  Publication  of  mappings  relating  CMMI  to  9001  and  12207  by 
Software  Quality  Institute, 

(2004)  Breakout  session  on  dual  outcome  SCAMPI  appraisals  at 
SCAMPI  Lead  Appraiser  workshop 

(2005)  Publication  of  Automotive  SPICE  (ASPICE)  -  a  derivative  of 
12207  for  use  by  automobile  manufacturers  to  select  suppliers, 

(2003-5)  Informal  SCAMPIs  with  9001 -relevant  outcomes  reported 
by  individual  SCAMPI  Lead  Appraisers 
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Synergy  With  Selected  International 
Standards  -1 

Identify  areas  where  there  are  opportunities  for  synergy 
between  key  international  standards  and  the  CMMI 
Product  Suite 

Exploit  these  opportunities  by  developing  appropriate 
work  products  and/or  liaising  with  appropriate 
individuals  and  organizations 
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Synergy  With  Selected  International 
Standards  -2 

Key  standards  identified  to  date  are 

•  ISO  900x:2000  family  of  standards  (as  well  as 
selected  domain  derivatives) 

•  ISO/I  EC  12207 

•  ISO/I  EC  15288 

•  ISO/I  EC  15504 

Note  that  15504  provides  a  mechanism  for  establishing 
important  relationships  to  other  important  process- 
related  international  standards 
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Carnegie  Mellon 

Software  Engineering  Institute  CMMI 

Revised  Approach  for  9001 

Modified  position  as  follows: 

*  Current  market  needs  are  for  9001 -relevant  outputs 
from  a  SCAMPI  as  opposed  to  a  15504  process 
profile  relevant  to  a  9001  process  reference  model, 

*  initial  pilots  will  not  focus  on  15504  conformance, 

*  Possibility  of  15504  conformance  relative  to  a  9001 
process  reference  model  as  market  needs  evolve 
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Usage  Scenarios 

Initial  pilots  this  year  and  next  are  focused  on 
addressing  the  following  scenarios: 

•  “we  are  also  working  towards  9001  compliance  - 
how  are  we  doing? 

•  “we  have  already  achieved  9001  certification  and 
we  would  like  to  see  how  our  [CMMI  initiative,  9001 
certification]  is  supporting  our  [9001 
accomplishments,  CMMI  initiative]” 

•  Variations  of  above: 

-  If  we  are  CMMI  maturity  level  x,  what  are  the 
implications  for  9001  certification? 

-  If  we  are  9001  certified,  what  are  the  implications 
for  CMMI  maturity? 


©  2005  by  Carnegie  Mellon  University 


Page  10 


Carnegie  Mellon 

Software  Engineering  Institute 


CMMI 


CMMI  -  ISO9001 :2000  Capability  Profile  by 

Equivalent  Staging 


Process  Areas 


^  CL  O  ^  ^ 

o  ^  ^  p  Q 

LU  Q_  CO  CL 

dc  Q- 

opoIcn-JLLQh-^^Q: 

DC  i —  lu<QlclOcl^< 

>  >  O  o  —  CO  Q 

DC 

WdO 

ddO 

OID 

CAR 

Maturity  Level  2 

Maturity  Level  3 

ML4 

ML5 

Courtesy  of  Rout  et  al,  QualCon  2004 
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The  possible  options  for 


assessment  and  surveillance 

Current  ISO  9001 

(Combined  ISO  Surveillance 

ISO  9001  1 - 

IA  , _ .  / 

SCAMPI  A’ 

using  Cat  C’  appraisal)  , 

& 

ISO  9001 

♦  ^  ♦  l 

Rating  letter  &  or  certificate 
with  scope  indicating 
"...  in  accordance  with  Level  X” 


Current  CMMI 


SCAMPI 

‘A’ 


Rating  letter 
indicating  level 
achieved 


continues  to 
demonstrate 
compliance  with 
ISO  9001:2000 


...no  behaviours 
inconsistent  with 
operating  at  level  X 


SCAMPI  A’ 


(Cat  ‘C’  appraisal) 
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Current  Status 

October  2005  -  “Shadow  appraisal”  as  an  initial  step 
towards  identifying  and  capitalizing  on  synergies  with 
9001, 

2005-6  -  Possible  collaboration  with  US  Tag  to  TCI  76 
for  development  of  guidance  document  useful  to  9001 
and  CMMI  communities, 

February  2006  -  Full  SCAMPI  appraisal  to  test  ability  to 
generate  9001  outcomes  as  well  as  a  12207  process 
profile  generated  in  compliance  with  15504-2, 

Technical  Note  (s)  to  document  lessons  learned  and 
provide  guidance 
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The  Application  of 
IEEE  Software  and  System 
Engineering  Standards 
in  Support  of 

Software  Process  Improvement 
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Northrop  Grumman  IT 
Huntsville,  AL 
uiiuufrJlund@ngc.com 
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In  Other  Words... 


Using  IEEE  Software  Engineering 
Standards  to: 


•  Define  software  engineering  (SE) 
processes. 

•  Ensure  CMMI-SW  Level  2 
compliance. 

•  Perform  software  engineering 

gap  analyses.  I 

•  Improve  existing  SE  processes^ 
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Why  Process  Improvement? 


•  All  those  practicing  as  software  engineers  should  desire  to 
evolve  out  of  the  chaotic  activities  and  heroic  efforts  of  a 
Level  1  organization. 


-  Because  no  one  likes  a  ‘painful’  work  environment  - 


•  Good  software  can  be  developed  by  a  Level  1 
organization,  but  often  at  the  expense  of  the  developers. 

-  People  get  tired  of  being  the  hero  - 


•  At  the  repeatable  level,  Level  2,  software  engineering 
processes  are  under  basic  management  control  and  there 
is  a  management  discipline. 

-  Even  the  most  die-hard  techie  needs  time  away  from  work  - 
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Who  Cares? 


TASC 


The  Organization 

♦  Interested  in  defining  sound  software  engineering 
practices. 


•  Would  like  to  perform  a  Gap  analysis  on  existing 
processes. 


•  Would  like  to  demonstrate  CMMI  Level  2  capability. 


The  Individual 


•  Tasked  to  implement  CMMI  compliant  processes. 

•  Would  like  to  improve  existing  software  engineering 
capabilities. 

•  Would  like  to  demonstrate  CMMI  Level  2  capability. 


NDIA  CMMI  Conference,  2005 
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•  IEEE  Standards  can  be  used  as  tools  to  help 
in  the  painful  process  of  ‘self-documentation’. 

•  Many  of  the  standards  provide  detailed 
procedure  explanations,  they  offer  section  by 
section  guidance  on  building  the  necessary 
documentation. 

•  Most  importantly,  they  provide  the  best 

practice  as  defined  by  those  from  the  software 

development  industry  who  sit  on  the  panels  of 
■ 
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information  Technology 


The  CMMI  and  SWE  Standards 


TASC 


The  CMMI  is  a  compendium  of  software 
engineering  practices,  which  act  as 
the  motivator  for  the  continuous  evolution 
of  improved  software  engineering  processes. 


IEEE  Standards  can  be  used  to 
provide  the  basic  beginning  framework 

for  software  process  improvement. 
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Software  and  Systems  Engineering  Standards 

Committee  (S2ESC) 


“To  provide  a  family  of  products  and  services  based  on  software 
engineering  standards  for  use  by  practitioners,  organizations,  and 
educators  to  improve  the  effectiveness  and  efficiency  of  their  software 
engineering  processes,  to  improve  communications  between  acquirers 
and  suppliers,  and  to  improve  the  quality  of  delivered  software  and 
systems  containing  software.  ” 

In  1996  and  1998  S2ESC  conducted  two  web-based  software 
engineering  users  surveys,  the  results  of  these  surveys  indicated  that 
users  perceived  the  standards  provided  the  most  value  when  applied  as 
guidance  in  support  of  software  process  improvement  efforts. 


http://standards.computer.org/sesc/ 
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•  Users  view  IEEE  software  engineering 
standards  primarily  as  reference  material  to 
develop  their  own  internal  plans. 

•  IEEE  SE  standards  are  tailored  and  used  to 
develop  internal  documentation  for  compliance 
measures,  namely  CMMI. 

•  There  is  value  added  in  the  use  of  the  IEEE 
software  engineering  standards  set  in  support 
of  process  improvement  activities. 
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My  Personal  Goals 


Show  how  the  IEEE  set  of  software  engineering 
standards  may  be  applied  to  facilitate 
CMM/CMMI  Level  2. 

Examine  Strengths  and  weaknesses 
of  each  standard  in  support  of  CMM 
Level  2  requirements. 

Provide  recommendations  on  how  the  I 
software  engineering  standards  set  may  most 
effectively  be  utilized  to  establish  software 
process  controls. 


NDIA  CMMI  Conference,  2005 
©2005  Susan  K.  Land.  All  rights  reserved. 


NORTHROP  GRUMMAN 

information  lechnoiogy 
TASC 


The  logic 


Assumption  1 


The  CMMI-SW  Staged  is  an  upgrade  of  the 
CMM. 


Assumption  2 


IEEE  Standards  proved  to  be  an  effective  support  for  the 
implementation  of  CMM-based  process  improvement. 


Therefore 


IEEE  Standards  provide  effective  support  for  the 
implementation  of  CMM  and  CMMISW-based  process 
improvement. 
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The  Basics  -  CMM/CMMI 


A  process  is  a  leverage  point  for  an 
organization’s  sustained  improvement 


•  The  purpose  of  the  CM  Ml  is  to  provide 
guidance  for  improving  processes  within  an 
organization. 


•  CMM  vl  .1  being  phased  out,  CMMI-SW 
builds  on  CMM  vl  .1  and  supports  integrated 
enterprise-wide  process  improvement 
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Overview  Comparison 


Maturity 

Level 

SW-CMM 

CMMI-SW 

(Staged) 

5 

Optimizing 

Optimizing 

4 

Managed 

Quantitatively 

Managed 

3 

Defined 

Defined 

2 

Repeatable 

Managed 
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CMMI-SW  (Staged)  Level  2  PAs 


Maturity  Level 

Process  Area  (PA)  Name 

#  of  Key 
Practices 

5 

Organizational  Innovation  and  Deployment 

19 

Optimizing 

Causal  Analysis  and  Resolution 

17 

4 

Organizational  Process  Performance 

17 

Quant.Managed 

Quantitative  Project  Management 

20 

3 

Requirements  Development 

20 

Defined 

Technical  Solution 

21 

Product  Integration 

21 

V  erifi  cation/ V  alidation 

20 

Organizational  Process  Focus 

19 

Organizational  Process  Definition 

17 

Organizational  Training 

19 

Integrated  Project  Management 

20 

Risk  Management 

19 

2 

Requirements  Management 

15 

Project  Planning 

24 

Managed 

Project  Monitoring  and  Control 

20 

Process  and  Product  Quality  Assurance 

14 

Configuration  Management 

17 

Supplier  Agreement  Management 

17 

Measurement  and  Analysis 

18 
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CMMI  -  Structural  Overview 


Maturity  Levels 


Process  Area  1 


Process  Area  2  Process  Area  n 


Specific  Goals^)  (^Generic  Goals^) 


Comm 


\ 


Common  Features 


tment 


to  Perform 


Ability 
to  Perform 


Directing 
Implementation 


Verifying 
Implementation 
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CMMI-SW  Level  2 /The  Specifics 


Requirements  Management.  Manage  requirements  associated  with  a  project 
and  identify  inconsistencies  between  the  requirements  and  the  project  plan 
and  associated  work  products. 


Project  Planning.  Planning  in  support  of  project  activities. 


Project  Monitoring  and  Control.  Processes  supporting  the  effective 
management  of  a  software  project. 

Process  and  Product  Quality  Assurance.  Activities  associated  with  software 
project  oversight. 


Configuration  Management.  Processes  in  support  of  the  definition,  control, 
review,  and  reporting  of  the  work  products  associated  with  a  software  project. 


Supplier  Agreement  Management.  Processes  supporting  the  acquisition  of 
products  from  suppliers  for  which  there  exists  a  formal  agreement. 

Measurement  and  Analysis.  Processes  supporting  the  development, 
maintenance,  and  implementation  of  software  project  measurement  activities. 
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CMMI  &  IEEE  Standards 


•  CMMI  -  Prescriptive  (What) 


s  Provide  guidance  for  improving  the  processes 
within  an  Organization 

•  IEEE  -  Descriptive  (How) 


s  To  provide  a  family  of  products  and  services 
based  on  software  engineering  standards  ... 


ndia cmmi  conference,  2005  a  Logical  pairing  w  reach  process  improvement  goals 
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•  The  standards  specify  format  and  content 
with  no  recommendation  of  the  exact 
techniques  to  be  used. 


The  standards  represent  industry  best 
practices  having  been  developed  by 
domain  experts  with  broad  expert 
consensus. 


•  The  standards  specify  the  minimum 
required  contents  for  each  CMMI  support 

document. 
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The  IDEAL®  Approach 


Learning 


Initiate 

Diagnose 

Establish 

Act 

Learn 


ChjfpdflriE? 

tw$n\  1 
Cflslrtd  5t#1ns 


D^vglgp  ^ 


ard 

Vtfidaffi 


Build 

Spor-soiif^ 


ihlratf 


7™ 

Prlnriilcs 


Stimulus  lor  mange 


Initialing 


Diagnosing 


Acting 


Establishing 


•  Developed  to  support  the  CMM/CMMI 

•  Serves  a  road  map  to  software  process  implementation 

and  improvement 
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Identify  a  group  of  people  who  are  given 
responsibility  and  authority  for  improving 
organizational  processes: 

•  Implementing  process  improvement  can  be  very  time- 
consuming,  depending  upon  the  scope  and  complexity  of  the 
effort. 

•  Expectations  for  each  team  member’s  time  commitments  and 
job  responsibilities  must  be  modified  accordingly  to  reflect  the 
new  responsibilities. 

•  This  commitment  should  reflect  time  budgeted  for  process 
definition  and  improvement  and  any  required  refresher  training. 

IEEE  software  engineering  standards  provide  valuable  support  to 
the  process  team.  The  standards  should  be  used  to  help  define  and 
document  the  initial  baseline  of  recommended  processes  and 
practices. 
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Set  Realistic  Goals  (Diagnose) 


The  leap  from  chaos  (Level  1 )  to  Level  2  is  often  the  hardest 
step  for  many  organizations. 


•  Defining  the  initial  process  baseline  is  key,  in  order  to 
understand  where  the  organization  needs  to  be;  it  must  first 
understand  where  it  is. 

•  Use  the  CMMI®-SW  Level  2  and  Level  3  goals  to  identify  areas 
of  weakness  or  bottlenecks  in  existing  processes.  Then  refer  to 
each  of  the  appropriate  IEEE  Software  Engineering  standards 
using  them  as  planning  tools  and  as  checklists  to  be  considered 
when  determining  how  to  accomplish  process  completeness. 

•  It  is  important  to  identify  which  organizational 
process  plans  will  be  developed  and  the 
sequence  of  their  development. 
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Fix  Timelines  (Establish) 


Goal  driven  process  improvement  is  the  most  effective.  Identify 
short  and  long  term  goals  and  time  periods;  associate  these 
goals  as  schedule  milestones. 


(0-3  months) 

•  Identify  responsible  individuals. 

•  Identify  participating  project  managers. 

•  Identify  candidate  projects. 

•  Solidify  backing  of  Senior  Management. 

•  Look  at  existing  processes. 

•  Define  the  formats  for  your  process  plans  using  IEEE  Software 
Engineering  Standards  and  measure  them  against  the  CMMI® 
requirements. 

•  Get  project  members  to  provide  feedback  on  process  plans, 
review  and  incorporate  feedback. 

•  Conduct  ARC  Class  C  Gap  Analysis. 
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(3-6  months) 

•  Create  process  document  templates  (e.g.,  Software 
Development  Plan,  Software  Requirements  Specification.) 

•  Conduct  weekly/monthly  status  reviews. 

(6-9  months) 

•  Conduct  CMMI®-based  reviews  of  the  projects. 

•  Provide  feedback  regarding  project  reviews. 

(9-12  months) 

•  Conduct  Internal  Assessments,  with  reporting  to  senior 
management. 

•  Provide  feedback  regarding  project  review  providing 
requirements  for  improvement  to  the  projects. 
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Use  IEEE  standards  to  develop  your  baseline  process 
documentation. 


•  Once  a  process  baseline  has  been  established  formulate  an 
action  plan. 

•  It  is  also  important  to  evaluate  and  identify  any  potential  tools 
that  may  be  used  in  support  of  process  automation: 

-  A  tool  is  not  a  substitute  for  a  process. 

-  An  ideal  candidate  area  for  this  type  of  automation  is  SCM. 

•  Many  IEEE  SWE  standards  provide  documentation  templates 
and  describe  in  detail  what  the  processes  should  contain. 

Think  of  these  standards  as  an  in-house  software  process 
consultant  who  has  recommended,  based  upon  years  of 
experience,  the  proper  methodologies  and  techniques  to  be 
used  in  support  of  software  development. 
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Perform  Gap  Analysis  (Learn) 


It  is  important  to  gauge  how  effectively  process 
improvements  have  been  implemented  for 
continuous  process  improvement  to  be 
successful. 


•  Develop  a  benchmarking  appraisal  to  support  gap 
analysis  activities. 

•  Provides  a  baseline  for  future  process  improvement 
efforts  and  will  identify  weaknesses  and  strengths. 


•  Review  the  associated  appraisal  methodology  used 
in  support  of  the  CMMI® 


Looking  at  the  Specifics.. 
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Requirements  Management 

IEEE  Std  830-  1998 

IEEE  Recommended  Practice  for  Software 
Requirements  Specifications 

Project  Planning 

IEEE  Std  1058 -  1998 

IEEE  Standard  for  Software  Project 
Management  Plans 

Project  Monitoring  and  Control 

IEEE  Std  1058 -  1998 

IEEE  Standard  for  Software  Project 
Management  Plans 

Process  and  Product  Quality 

Assurance 

IEEE  Std  730  -  2002 

IEEE  Standard  for  Software  Quality 
Assurance 

Configuration  Management 

IEEE  Std  828  -  1998 

IEEE  Standard  for  Software  Configuration 
Management  Plans 

Supplier  Agreement  Management 

IEEE  Std  1062 -  1998 

IEEE  Recommended  Practice  for  Software 
Acquisition 

Measurement  and  Analysis 

IEEE  Std  1045  -  2002 

IEEE  Standard  for  Software  Productivity 
Metrics 
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•  Software  Life  Cycle 

-  IEEE/EI A  12207.0,  Industry  Implementation  of  International 
Standard  I  SO/I  EC 1 2207:1 995  — Standard  for  Information 
Technology  — Software  life  cycle  processes 

•  IEEE/EI  A  1 2207.1 ,  Industry  Implementation  of  International  Standard 
ISO/IEC1 2207:1 995  —  (ISO/IEC  12207)  Standard  for  Information 
Technology  — Software  life  cycle  processes  -  Life  Cycle  Data 

•  IEEE/EI  A  12207.2,  Industry  Implementation  of  International  Standard 
ISO/IEC1 2207:1 995  —  (ISO/IEC  12207)  Standard  for  Information 
Technology  — Software  life  cycle  processes  -  Implementation 
considerations 


•  Systems  Life  Cycle 

-  ISO/IEC  15288,  Systems  engineering  — 
System  life  cycle  processes 


www.  computer.  org/Standards 
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IEEE  Std  830-1998,  IEEE  Recommended  Practice  for 
Software  Requirements  Specifications. 

Outlines  the  requirements  for  what  comprises  a  good  Software 
Requirements  Specification  (SRS): 

•  Establishes  the  basis  for  agreement  between  the  customers  and 
the  suppliers  on  what  the  software  product  is  to  do. 

•  Reduces  the  development  effort. 

•  Provides  a  basis  for  estimating  costs  and  schedules. 

•  Provides  a  baseline  for  validation  and  verification. 

•  Facilitates  transfer. 

•  Serves  as  a  basis  for  enhancement. 


Does  not  directly  address  Requirements  Traceability! 


NDIA  CMMI  Conference,  2005 
©2005  Susan  K.  Land.  All  rights  reserved. 


•  Specifies  a  suggested  format  for  a  project  management  plan: 

-  This  document  may  be  used  as  a  guide  for  documenting  the  practices 
and  procedures  unique  to  each  organization  for  all  types  of  software 
efforts. 


-  The  IEEE  Standard  for  Project  Management  Plans  can  be  used  as  a 
model  for  this  CMMI  Level  2  process. 


•  The  purpose  of  CMMI  Level  2  Software  Project  Planning  is  to 
establish  reasonable  plans  for  performing  software  engineering  and 

software  project  management. 
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Simply  initially  estimating  the  duration  and  total  cost  of  a 
software  effort  is  not  sufficient. 

•  Planning  must  continue  throughout  the  software  development 
and  maintenance  process. 

•  Project  monitoring  (tracking)  and  control  of  the  management 
process  encompasses  most  of  the  development  process. 

This  includes  all  activities  that  project  management  has  to 
perform  to  ensure  that  the  project  objectives  are  met  and 
that  development  proceeds  according  to  the  plan. 

•  Monitor  cost,  schedule,  quality,  and  potential  risk. 

•  Take  corrective  action  when  necessary. 
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IEEE  Recommended  Practice  for  Software  Acquisition, 
IEEE  Std  1062- 1998. 


-  Provides  information  on  the  recommended  practice  for 
acquiring  software: 

•  Describes  the  software  acquisition  life  cycle. 

•  Offers  support  in  preparing  contract  requirements,  proposal 
evaluation,  and  supplier  selection. 

•  Provides  insight  into  the  management  of  a  software  supplier  and 
product  acceptance. 

•  Offers  a  series  checklists  which  consist  of  information  designed 
to  help  organizations  establish  their  own  software  acquisition 
process. 


■  This  standard  describes  a  set  of  quality  practices  that  can  be  applied 
during  one  or  more  steps  of  the  software  acquisition  process. 

NDIA  CMMI  Conference,  2005 
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•  The  purpose  of  IEEE  Std  730-1 998  is  to  provide 
uniform,  minimum  acceptable  requirements  for  the 
preparation  and  content  of  Software  Quality  Assurance 
Plans: 

•  Recommended  approaches  to  good  SQA  practices  are 
describe  in  IEEE  Std  730.1-1995. 


Combined,  these  two  plans  describe  the  requirements  in  support 


of  industry  standard  SQA  practices. 
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PA  -  Configuration  Management 


SCM  as  described  by  IEEE  Std  838-1998: 


“SCM  constitutes  good  engineering  practice  for  all  software  projects, 
whether  phased  development,  rapid  prototyping,  or  ongoing  maintenance. 
It  enhances  the  reliability  and  quality  of  software  by  providing  a  structure 
for  identifying  and  controlling  documentation,  code,  interfaces,  and 
databases  to  support  all  life  cycle  phases  supporting  a  chosen 
development/maintenance  methodology  that  supports  the  requirements, 
standards,  policies,  organization,  and  management  philosophy  producing 
management  and  product  information  concerning  the  status  of  baselines, 
change  control,  tests,  releases,  audits,  etc.” 


The  plan  basically  provides  a  framework  for 
organizations  to  follow.  Use  of  this  standard  offers  a 
reasonably  stable  cross-project  development 

environment. 
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•  IEEE  Std  1044.  Standard  Classification  for  Software 
Anomalies. 

-  Defines  a  uniform  approach  to  the  classification  and  documentation  of 
the  variances  found  in  software  products. 


•  IEEE  Std  1045.  Standard  for  Software  Productivity 
Metrics. 

-  Provides  a  framework  for  measuring  and  reporting  software 

productivity.  It  is  meant  for  those  who  want  to  measure  the  productivity 
of  the  software  process  in  support  of  their  software  product. 


Through  the  application  of  these  standards  -  issues 
with  life  cycle  processes  are  identified  and  improved . 


■ 
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Examine  each  CMMI  Level  2  Key  Practice  (Co,  Ab,  Me,  Ve,  and  Ac). 

Identify  supporting  portions  of  IEEE  standards.  Do  not  consider  each 
standard  in  isolation,  rather  consider  the  complete  set  of  those  most 
directly  supporting  CMMI  Level  2  items. 

Document  your  processes  using  the  IEEE  standards  and  Level  2 
capabilities. 

•  Small  projects  may  require  less  formality  in  planning  than 
large  projects,  but  all  components  of  each  standard  should 
be  addressed  by  every  software  project. 

•  Components  may  be  included  in  the  project  level 
documentation,  or  they  may  be  merged  into  a  system-level 
or  business-level  plan,  depending  upon  the  complexity  of 
the  project. 
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overly  prescriptive 

•  Remaining  confined  to  a  specific 
stage 

•  Lack  of  incentives 

•  No  metrics  taken 

•  Documentation  for  the  sake  of  documentation 
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What  to  watch  out  for. 


Each  organization  using  IEEE  standards  should  develop  a 
set  of  practices  and  procedures  that  provide  detailed 
guidance  for  preparing  and  updating  plans  based  upon 
standards. 


•  There  are  some  holes  relating  to  PT&O  and  metrics. 

•  Pay  special  attention  to  CMMI  general  requirements. 


Funding  for  process  improvement  activities  is  not 
specifically  referenced  in  IEEE  plans,  this  must  be 
included  in  the  project  management  plan. 

Need  to  specifically  address  requirements 
traceability  throughout  product  lifecycle. 
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•  Leverage  the  expertise  contained  in  the  IEEE  Software 
and  Systems  Engineering  Standards. 


•  Fix  timelines  to  produce  goal  driven  process 
improvement. 

•  Define  your  processes  in  outline  form. 

•  Perform  a  gap  analysis. 

•  Redefine  your  processes. 

•  Use  IEEE  standards  to  develop  your  baseline  process 
documentation. 


•  Perform  self-audit  using  CMMI  PAs. 

•  Readjust  processes/plans  based  upon  audit  results. 


Make  a  plan.  Then  follow  the  plan.  -  Watts 

Humphrey 
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Get  Involved 


IEEE  Computer  Society: 
http://www.computer.org/ 


IEEE  Software  Engineering  Standards: 
http://standards.computer.org/sesc/ 

IEEE  Software  Engineering  Online: 
http://billing.computer.org/portal/index.jsp 


CMM/CMMI: 

http://www.sei.cmu.edu 

To  Order  IEEE  Standards  ($320): 
http://www.computer.org/cspress/CATALOG/st01 1 21  .htp 
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Need  More  Help? 


S.  Land,  Jump  start  CMM/CMMI  Software  Process 
Improvement/Using  IEEE  Software  Engineering 
Standards,  John  Wiley/IEEE  Press,  Feb  2005. 


S.  Land,  J.  Walz,  Practical  CMMI  Software  Project 
Documentation/Using  IEEE  Software  Engineering 
Standards,  John  Wiley/IEEE  Press,  Oct  2005. 


IEEE  Software  Engineering  Standards  Collection, 
Institute  of  Electrical  and  Electronics  Engineers,  Inc. 
New  York,  NY,  2003. 

CMMI®  -SE/SW/IPPD/SS,  VI.  I,  Carnegie  Mellon 

University,  Software  Engineering  Institute,  Pittsburgh, 
PA.  March  2002. _ 
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Implementing  Process  Improvement 


Does  Size  Matter 

or  Was  Yoda 

Right? 


Yoda 


With  the  Force 
size  matters 
not,  do  or  do 
not,  there  is 
no  try! 


The  Question 


Does  the  size  of  the  organization 
change  any  of  the  fundamentals 

associated  with  the 
implementation  of  CMMI? 


Agenda 


The  Fundamentals 

Application  to  Large  Organizations  (>=  500) 
and  Medium  Organizations  (100-500) 

Application  to  Small  Organizations  (1-1 00) 

Conclusions 


Fundamentals 


Which  of  these  are  drivers? 

■  Need  to  Change 

■  Costs 

■  Competitive  requirement 

■  CEO/CIO  attended  seminar  where  CMMI  was 
mentioned 


Fundamentals 


ANSWER: 
ALL  ARE 


Fundamental  -  Change 


It  is  not  necessary  to  change. 
Survival  is  not  mandatory. 


W.  Edwards  Denting  (1900-1993) 


Fundamental  -  Change 


Fundamental  -  Change 


Less  than  30%  of  those  companies  that  were  in  the 
Fortune  500  in  1980  remain  in  the  Fortune  500 
today. 

Their  success  became  their  failure  because  they 
didn’t  see  the  need  to  change. .  .to  adapt  to  a  new 
world  order... until  it  was  too  late. 

There  is  no  “status  quo.”  That  is  an  illusion.  You 
are  either  getting  better  faster  than  the  competition, 
or  you  are  getting  worse  faster. 


Fundamental  -  Expense* 


175,000  IT  projects  are  attempted  annually  at  a  cost 
of  more  than  $250  billion 

Over  31%  of  all  projects  are  canceled  at  a  cost  of 
$81  billion 

Over  50%  of  all  projects  exceed  their  original 
estimates  by  almost  1 00% 

Rework  is  40%  or  more  of  the  cost  of  software 
development  projects 


Source:  The  Standish  Group  CHAOS  Report 

*  Projects  Only/Does  not  include  Maintenance  or  Enhancement  efforts 


Fundamental 


Federal  Contracts 
State  Contracts 
Local  Contracts 
Banking 
Pharmaceuticals 
Automotive 


Requirement 


Fundamental  -  CEO/CIO 


Without  management  support, 
you  are  finished  before  you 

start 


Fundamentals 


Infrastructure  of  successful  Process 
Improvement  efforts: 

■  Masochistic  Target  of  Opportunity  (MTO),  AKA 
“The  Process  Person”, 

■  Sponsor  -  preferably  one  with  authority  and 
spine 

■  Money  -  training,  reference  materials, 
newsletter,  trinkets,  etc. 

•  Hint  -  quality  IS  NOT  free 


Fundamentals 


Infrastructure  of  successful  Process 
Improvement  efforts: 

■  Plan  -  how  to  “Git  er  don” 

■  Acolytes  and  Test  Subjects  -  MTO’s  in  training 
and  pilot  projects 

•  Without  Acolytes  you  are  finished  after  you  start 

■  Incentives  -  two  approaches 

•  Beatings  will  continue  until  moral  improves 

•  Reward  system  for  achievement  that  covers  more  than 
the  CIO 


Fundamentals  -  Infrastructure 


Fundamentals  -  Infrastructure 


♦  Management  Steering  Committee 

■  Provides  executive  leadership,  support,  and  guidance 

♦  Engineering  Process  Group-  Composed  of 
process  area  leads  and  senior  managers 

■  Provides  operational  direction,  supervision  and  leadership, 
technical  guidance,  and  process  stewardship 

♦  Process  Action  Teams  -  Composed  of 
Process  Leads  and  Process  Team  Members 

■  Accomplish  process  development,  mentoring,  and  deployment 


Large/Medium  Organizations 
(>=500)  &  (100-500) 


In  general,  anyone  NOT  seen  these 
fundamentals  apply? 

■  Single  division/business  units 

■  Corporate  across  multiple  divisions/business 
units 

■  Projects  within  divisions/business  units 

■  Departments  within  divisions/business  units 


Small  Organizations 
(1-100) 


Do  these  fundamentals  apply? 

■  Change - 

•  90%  of  small  business  fail  -  one  trick  pony? 

•  For  a  small  team  in  a  large/medium  organization  - 
single  focus  or  general  (robotics/development)  - 
technology  drive 

■  Costs 

•  Small  business  -  cash  is  king  (see  failure  stat) 

•  Small  team  -  can  you  say  “overseas  or  outsourced” 


Small  Organizations 
(1-100) 


Do  these  fundamentals  apply? 

■  Competitive  requirement  - 

•  Small  business  -  SBA  programs  and  set-asides 

•  Small  team  -  Other  teams  want  you,  consolidation 

-  CEO/CIO  Support 

•  Small  business  -  much  closer  to  the  problems,  but  less 
latitude  to  solve  them 

•  Small  team  -  don’t  have  CEO/CIO  but  managers  are 
expected  to  think  like  them 


Small  Organizations 
(1-100) 


Do  these  fundamentals  apply? 

.  MTO 

•  Small  business,  Small  team  -  who’s  responsible 

■  Sponsor 

•  Small  business,  Small  team  -  both  need  buy-in 

■  Money  -  nuff  said 


Small  Organizations 
(1-100) 


Do  these  fundamentals  apply? 

■  Plan 

•  Small  business,  Small  team  -  try  without  one 

■  Acolytes  and  Test  Subjects 

•  Small  business,  Small  team  -  need  buy-in  from  doers 

■  Incentives 

•  Small  business,  Small  team  -  only  time  that  “keep 
your  job”  may  actually  be  single  reason 


Conclusion 


Understanding  that  Scope  of  Work  and  Level 
of  Effort  to  accomplish  the  work  are  not  the 
same  as  Fundamental  principles  that  apply  to 
the  work  -  Size  does  not  matter! 

■  Build  an  overpass,  build  the  Golden  Gate  Bridge 


Counter  Opinions? 


Speak  now  or  I  will  assume  you  are: 

■  In  agreement 

■  Assimilated 

■  Numbed  by  information  overload 

■  Don’t  care  -  its  Reception  time 


Process  Improvement  should  be  the 
light  at  the  end  of  the  tunnel,  not  a 
train  coming  at  you 


Contact  Information 

Paul  H.  Meyers, 

SAIC  Process  Improvement  Manager 

301-763-3911 

paul.h.meyers@saic.com 


Business 
Transformation 
Institute 


CarncgieJYleUon 

Software  Engineering  institute 


Building  a  Credible  SCAMPI  Appraisal 

Representative  Sample 


Bob  Moore,  Business  Transformation  Institute,  Inc. 
Will  Hayes,  Software  Engineering  Institute 
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What  is  Design  of  Experiments? 


•  Design  of  Experiments  (DOE)  is  a  mathematical 
statistics  technique  used  to  help  understand  the 
influence  that  different  experimental  factors 
have  on  the  response  from  a  system. 

-  DOE  allows  us  to  understand  the  interaction  between 
factors,  as  opposed  to  experimentation  that  changes 
just  one  factor  at  a  time. 

-  DOE  provides  a  means  for  maximizing  the 
information  gained  from  each  experiment,  thus 
reducing  the  number  of  experiments  that  we  need  to 
conduct. 
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DOE  in  SCAMPI 


•  DOE  has  two  applications  for  SCAMPI  A,  B, 
and  C  appraisals: 

1.  Appraisal  Planning:  DOE  can  help  to  construct  an 
appropriate  representative  sample  of  the 
organizational  unit  (OU)  to  be  appraised. 

2.  Appraisal  Execution:  DOE  can  help  to  choose 
which  personnel  should  be  interviewed  and  which 
questions  should  be  asked  in  collecting  affirmations. 
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Why  Should  We  Care  About 
A  Good  Representative  Sample? 


•  A  well-constructed  representative  sample  leads  to  a  superior 
appraisal  return  by: 

-  Selecting  for  examination  the  set  of  instantiations  that  provide  the 
greatest  potential  for  verifying  process  institutionalization  per  each 
member  of  the  examined  set  of  instantiations. 


•  This  provides  the  most  information  gained  per  appraisal  resource  invested. 

•  Other  sets  of  instantiations  could  be  examined,  but  would  be  inferior  with 
respect  to  insights  gained  on  process  institutionalization. 

-  Enhancing  the  credibility  of  the  appraisal  by  providing  defensible 
reasoning  that  led  to  the  selection  of  some  instances  to  be  included  in 
the  appraisal  while  excluding  others: 

•  A  representative  sample  that  excludes  some  instantiations  without  clear 
reason  invites  suspicion  that  the  appraisal  results  may  not  reflect  OU 
process  institutionalization  because  instantiations  detrimental  to  the  OU's 
case  for  institutionalization  are  being  avoided. 

•  Likewise,  a  representative  sample  that  insists  on  including  some 
instantiations  without  justification  might  raise  questions  about  the  appraisal 
results  again,  only  this  time  because  instantiations  that  reflect  atypical 
"good"  institutionalization  effort  are  being  included. 
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How  are  Representative  Samples 
Constructed  Now? 


•  The  SCAMPI  Method  Description  Document  does  not  give  us  much 
advice! 

-  "Upon  determining  that  sufficient  coverage  of  the  reference  model  and 
the  organizational  unit  has  been  obtained,  appraisal  findings  and 
ratings  may  be  generated."  (SCAMPI  MDD,  p.  1-11.) 

-  Coverage  is  said  to  imply: 

•  "  (a)  the  collection  of  sufficient  data  for  each  model  component  within  the 
CMMI  reference  model  scope  selected  by  the  sponsor,  and" 

•  "(b)  obtaining  a  representative  sample  of  ongoing  processes)  spanning  the 
life-cycle  phases  that  the  appraised  organization  is  using  in  the  development 
and  delivery  of  its  products  and  services." 

-  The  lead  appraiser  is  further  cautioned  to  construct  a  "valid  sample  of 
the  organizational  unit  to  which  results  will  be  attributed"  based  on 
organization  size,  scope,  and  geographic  dispersion. 

-  The  lead  appraiser  and  sponsor  are  reminded  that  all  statements  should 
be  accurate  in  describing  the  organization  to  which  results  may  be 
attributed. 
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•  Given  this  guidance,  how  is  a  lead  appraiser  to  construct 
a  "valid  sample"  that  can  withstand  rigorous, 
independent  examination? 

•  The  current  typical  practice  of  using  no  more  than  four 
projects  in  an  appraisal,  no  matter  the  size  of  the 
appraised  organizational  unit,  may  entirely  miss 
information  that  characterizes  how  well  or  poorly  the 
OU  is  doing  with  its  processes. 

•  Unfortunately,  increasing  the  number  of  projects 
examined  doesn't  help! 

-  Very  large  samples  of  projects  from  a  large  OU  soon  become  cost 
prohibitive  without  providing  analytically  defensible  insight 
into  process  performance 

-  Although  saying  "we  looked  at  10  projects  and  10,000  artifacts" 
sounds  impressive  —  even  if  it  isn't! 
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But  What  Else  Can  We  Do? 


•  Since  a  SCAMPI  A  appraisal  is  meant  to  provide 
a  benchmark  of  an  OU's  process  performance, 
we  need  some  technique  that: 

-  Seeks  to  maximize  information  received, 

-  Minimizes  cost,  and 

-  Provides  appropriate  rigor  to  justify  our  appraisal 
planning  choices  to  an  independent  examiner. 

•  DOE  provides  exactly  these  capabilities! 
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DOE  Language  and  SCAMPI 


Experiment  =  an  appraisal 

Experimental  factors  =  characteristics  of  the  OU  as  they  are 
observed  across  different  parts  of  the  organization  where  work  is 
underway 

Experimental  design  =  the  list  of  instantiations  from  which  we  will 
examine  artifacts,  based  on: 


-  The  experimental  factors  present  in  the  OU, 

-  The  budget  available  for  the  appraisal,  and 

-  The  amount  of  confounding  between  factors  we  are  willing  to  accept. 

•  Response  variables  =  weakness  and  strengths  of  process  area 
specific  or  generic  practices  and  satisfaction  of  goals. 

•  Factors  effects  =  the  influence  that  different  factors  have  on  the 
response  variables  under  consideration. 

•  Confounding  =  our  inability  to  distinguish  between  the  influence  on 
the  response  variables  of  one  or  more  factors  with  respect  to  another 
set  of  factors.  Confounding  is  undesirable,  but  may  be  managed 
through  choice  of  designs. 
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DOE  Language  and  SCAMPI,  Continued 


•  Replication  =  examining  more  than  one  instantiation  corresponding 
to  a  particular  set  of  experimental  factors  in  our  chosen  design— 
which  provides  better  insight  into  institutionalization  by  having 
additional  instantiations  to  confirm  observed  responses. 

•  Balanced  design  =  a  fractional  factorial  design  in  which  an  equal 
number  of  trials  (at  every  level  state)  is  conducted  for  each  factor. 

•  Block  &  Blocking  =  When  structuring  appraisals,  blocking  may  be 
used  to  account  for  some  unknown  that  one  wishes  to  avoid;  a  block 
may  be  a  dummy  factor  that  does  not  interact  with  the  real  factors. 

•  Orthogonal  =  An  appraisal  is  orthogonal  if  the  effects  of  any  factor 
balance  out  (sum  to  zero)  across  the  effects  of  the  other  factors. 
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•  Experimental  resolution  helps  us  to  understand  the 
degree  of  our  "known  unknowns"  in  an  appraisal. 

-  Resolution  I  =  we  gain  no  insight  from  an  appraisal 

-  Resolution  II  =  we  cannot  tell  the  difference  between  the 
influence  of  main  factor  effects  (why  bother?) 

-  Resolution  III  =  Main  factor  effects  are  confounded  (aliased) 
with  two-factor  interactions. 

-  Resolution  IV  =  No  main  factor  effects  are  aliased  with  two- 
factor  interactions,  but  two-factor  interactions  are  aliased  with 
each  other. 

-  Resolution  V  =  No  main  effect  or  two-factor  interaction  is 
aliased  with  any  other  main  effect  or  two-factor  interaction,  but 
two-factor  interactions  are  aliased  with  three-factor  interactions. 
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Example  OU  Experimental  Factors 


•  The  factors  that  influence  process 
institutionalization  in  an  OU  depend  on  that  OU. 
Some  typical  factors  to  be  considered: 

-  The  size  of  the  project: 

•  Projects  that  are  large  or  small  with  respect  to  the  OU's 
typical  project  mixture  may  influence  how  processes  are  used. 

-  Project  age: 

•  New  or  existing  projects  for  the  OU  may  have  different 
understanding  or  maturity  of  processes. 

-  Project  geographic  location: 

•  Projects  performed  at  a  core  location  or  at  a  remote  site  may 
differ  in  their  processes. 
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Example  OU  Experimental  Factors, 

Continued 


-  Project  dispersion: 

•  Projects  that,  within  the  context  of  the  project,  are  executed  at  one 
location  or  multiple  locations  that  are  inconvenient  for  daily  face-to- 
face  contact  may  have  different  processes. 

-  Project  parent  organization: 

•  The  "home"  or  sponsoring  OU  for  a  project  may  influence  how 
processes  are  implemented  depending  on  the  support  of 
management  for  the  processes. 

-  Project  complexity: 

•  Projects  that  have  complex  life  cycles  may  have  different  processes 
than  simpler  projects  (e.g.,  spiral  versus  waterfall  life  cycle). 

-  Project  customer  and  users: 

•  Projects  performed  for  different  customers  or  users  may  use 
different  processes  depending  on  the  customer  or  user's 
requirements. 
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How  to  Select  a  Representative 
Sample  for  an  Appraisal  (1) 


1.  Determine  the  objectives  of  the  appraisal  with  respect  to  the  OU 
scope  and  process  areas  to  be  considered. 

2.  List  the  factors  that  may  influence  process  institutionalization  in 
theOU. 


Be  generous  in  listing  factors  —  a  factor  that  has  no  real  impact  is 
easily  discarded  through  the  application  of  DOE  techniques,  but 
omitting  a  factor  of  real  influence  may  skew  the  appraisal 
conclusions. 


3.  Determine  if  any  of  the  factors  are  clearly  dependent  on  other 
factors.  If  so,  these  factors  may  be  collapsed  into  fewer  combined 
factors. 


4.  Determine  the  level  settings  for  the  factors,  such  as  project  size 
equals  one  of  "large"  or  "small".  Any  given  factor  may  have 
multiple  levels,  although  two  levels  are  easiest  from  a  design  and 
analysis  perspective. 

5.  List  all  of  the  instantiations  in  the  OU  that  are  supposed  to  be 
using  processes  corresponding  to  the  process  areas  under 
consideration. 
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How  to  Select  a  Representative 
Sample  for  an  Appraisal  (2) 


6.  For  each  instantiation  in  the  list,  determine  the  factor 
level  settings  that  describe  that  instantiation. 

•  For  example,  project  X  may  have  factor  levels  of  size=large, 
location=central  office,  and  duration=long,  where  as  project  Y 
may  have  factor  levels  of  size=small,  location=field  site,  and 
duration=short. 


7.  Given  the  list  of  factors  and  their  level  settings,  choose 
an  experimental  design. 

•  This  design  will  be  determined  by  how  much  confounding 
between  factors  is  tolerable  and  the  budget  limits  on  how 
many  different  instantiations  can  be  examined  in  the  appraisal. 

•  A  design  catalog  or  statistical  software  that  supports  DOE  is 
indispensable  here  for  exploring  the  options! 

8.  Fill  in  the  experimental  design  from  step  7  with  actual 
instantiations  using  the  information  in  step  6. 
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An  Example  of  Selecting 
A  Representative  Sample  Using  DOE 


•  Suppose  we  are  examining  an  OU  that  has  five 
factors  to  be  accounted  for  in  an  appraisal: 

-  Project  size:  large  or  small 

-  Project  age:  new  or  existing 

-  Project  geographic  location:  domestic  or  international 

-  Project  customer:  government  or  commercial 

-  Project  complexity:  high  or  low 

•We  have  5  factors  at  2  levels  that  might  influence 
process  institutionalization  in  the  OU. 
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Full  Factorial  Design 


•  The  full  factorial  design  (all  factors  at  all  levels),  we 
would  have  to  examine  32  (2x2x2x2x2) 
instantiations! 


•  The  design  is  given  on  the  next  page  for  illustration 
purposes. 

•  No  one  is  expected  to  ever  construct  such  an 
appraisal. 
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Size 

Age 

Location 

Customer 

Complexity 

Instantiation  =  1 

L  =  large 

N  =  new 

D  =  domestic 

G  =  government 

H  =  high 

2 

L 

N 

D 

G 

L  =  low 

3 

L 

N 

D 

C  -  commercial 

H 

4 

L 

N 

D 

C 

L 

5 

L 

N 

I  =  international 

G 

H 

6 

L 

N 

I 

G 

L 

7 

L 

N 

I 

C 

H 

8 

L 

N 

I 

C 

L 

9 

L 

E  =  existing 

D 

G 

H 

10 

L 

E 

D 

G 

L 

11 

L 

E 

D 

C 

H 

12 

L 

E 

D 

C 

L 

13 

L 

E 

I 

G 

H 

14 

L 

E 

I 

G 

L 

15 

L 

E 

I 

C 

H 

16 

L 

E 

I 

C 

L 

17 

S  =  small 

N 

D 

G 

H 

18 

S 

N 

D 

G 

L 

19 

S 

N 

D 

C 

H 

20 

S 

N 

D 

C 

L 

21 

s 

N 

I 

G 

H 

22 

s 

N 

I 

G 

L 

23 

s 

N 

I 

C 

H 

24 

s 

N 

I 

C 

L 

25 

s 

E 

D 

G 

H 

26 

s 

E 

D 

G 

L 

27 

s 

E 

D 

C 

H 

28 

s 

E 

D 

C 

L 

29 

s 

E 

I 

G 

H 

30 

s 

E 

I 

G 

L 

31 

s 

E 

I 

C 

H 

32 

s 

E 

I 

C 

L 
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Alternatives  to  Full  Factorial 


•  Except  in  limited  circumstances,  a  full  factorial 
selection  on  instantiations  is  too  expense  and 
too  time  consuming. 

-  Note:  for  this  presentation,  we  are  neglecting  the  idea  that  an 

appraisal  might  want  to  look  at  more  than  one  instantiation  for  each 
setting  of  factors  (replication).  Looking  at  multiple  instantiations  for 
the  same  factors  is  a  good  idea— but  the  number  of  instantiations  to 
be  examined  grows  rapidly! 

•  Besides,  who  needs  complicated  math  to  try 
every  combination  of  everything? 

•  DOE  offers  an  alternative:  fractional  factorial 
designs. 
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A  V4  Fractional  Factorial  Design 


•  In  the  example  above,  instead  of  using  a  full  factorial 
design,  we  could  also  have  conducted  our  appraisal 
using  a  fractional  factorial  design  of2f^=8 
instantiations.  1 

Number  of  j  Fraction 

Levels  of  Number  of 


Factors 


Factors 


•  A  V4  design  in  this  case  is  a  Resolution  III 
experiment. 

•  The  choice  of  a  fractional  factorial  design  will 
depend  on  the  number  of  factors  to  be 
considered  and  the  acceptable  experimental 
resolution. 
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The  V4  Fractional  Factorial  Design 


Instantiation 

Size 

Age 

Location 

Customer 

Complexity 

1 

S 

E 

I 

G 

H 

2 

L 

E 

I 

C 

L 

3 

S 

N 

I 

C 

H 

4 

L 

N 

I 

G 

L 

5 

S 

E 

D 

G 

L 

6 

L 

E 

D 

C 

H 

7 

S 

N 

D 

C 

L 

8 

L 

N 

D 

G 

H 
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Still  Too  Many  Instantiations! 


•  From  the  viewpoint  of  a  SCAMPI  A  appraisal, 
using  8  instantiations  across  multiple  process 
areas  still  seems  like  a  lot! 


-  Note:  we're  still  doing  better  than  a  traditional 
representative  sample  selection  method — we  at  least 
clearly  understand  the  impact  of  different  factors  on 
our  appraisal. 

•  Going  to  a  1  /  8  fractional  factorial  would  give  4 
instantiations  to  be  appraised — but  drops  our 
resolution  down  to  Resolution  II. 


•  What  to  do  ...  ? 
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Using  DOE  with  SCAMPI  B  and  C 


•  DOE  works  best  not  as  a  single  experiment,  but 
as  a  sequence. 


•  This  is  ideally  suited  to  SCAMPI: 

-  Conduct  early  appraisals  that  examine  many  factors 
and  instantiations  as  SCAMPI  Cs. 


-  Based  on  the  results,  eliminate  factors  (and  the  need 
for  instantiations) . 

-  Conduct  later  appraisals  that  examine  fewer  critical 
factors  and  instantiations  as  SCAMPI  As  or  Bs. 


-  Note:  changing  lead  appraisers  from  one  appraisal  to 
another  allows  you  to  block  your  design  according  to 
lead  appraiser  — if  lead  appraisers  are  unbiased! 
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Example  SCAMPI  C 


•  Consider  the  setup  in  the  example  above:  5  factors  that 
may  influence  the  OU's  process  institutionalization. 

•  We  would  like  to  determine  which  factors  really 
influence  the  process  and  which  are  not  important. 

•  Eventually,  we  want  to  benchmark  the  OU  using  a 
SCAMPI  A. 


•  We  will  start  with  a  SCAMPI  C. 


•  For  illustration  purposes,  we  will  only  look  at  the 
appraisal  covering  two  process  areas  (PP  and  REQM)  at 
Capability  Level  2. 

-  The  example  could  be  expanded  to  as  many  PAs  as  we  like,  but 
the  calculations  are  lengthy  in  a  presentation. 
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Assigning  Numerical  Values  to  Response 

Variables 


•  As  defined  in  the  SCAMPI  C  method,  we  would  usually  assign  a 
color  (green,  yellow,  red)  as  the  characterization  of  each 
instantiation's  specific  and  generic  practices. 

•  To  aid  in  our  analysis,  we  will  assign  numerical  values  against  these 
characterizations: 

-  Red  =  0 

-  Yellow  =  0.5 


-  Green  =  1.0 


•  The  assigned  values  may  be  changed,  if  desired. 

•  Similar  values  may  be  assigned  for  characterizations  using  in  other 
SCAMPIs.  For  example: 

-  NI  =  0 


-  PI  =  0.5 


-  0  =  0.75 


-  FI  =  1 
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Aggregation  at  the  Goal  Level 


•  To  aid  in  our  analysis,  we  are  going  to  take  the 
arithmetic  mean  of  the  specific  practices  and  generic 
practices  at  the  goal  level  for  each  instantiation. 

•  For  REQM  (similarly  for  PP), 

Score(SG  1)  = 

Score(SPl .  1  - 1)  +  Score(SPl  .2  -  2p-Score(SP\  .3  - 1)  +  Score(SP\  .4  -  2)  +  Score(SPl  .5 


-1) 


5 

Score  (GG  2)  = 

Score(GP2. 1)  +  Score(GP2.2 )  H - b  Score(GP2. 10) 

10 
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Institute 


We  perform  the  SCAMPI  C  appraisal  using  the 
instantiations  given  above. 

The  results: 


S 

A 

L 

C 

C 

PP 

SGI 

PP 

SG2 

PP 

SG  3 

PP 

GG  2 

REQM 

SGI 

REQM 
GG  2 

Instantiation  =  1 

s 

E 

I 

G 

H 

0.38 

0.71 

0.83 

0.75 

0.60 

0.75 

!  2 

L 

E 

I 

C 

L 

0.25 

0.86 

0.83 

0.75 

0.60 

0.75 

3 

S 

N 

I 

C 

H 

0.88 

0.93 

1.00 

0.95 

0.80 

0.95 

4 

L 

N 

I 

G 

L 

0.88 

0.86 

1.00 

0.95 

0.90 

0.95 

5 

S 

E 

D 

G 

L 

0.13 

0.14 

0.17 

0.20 

0.10 

0.20 

6 

L 

E 

D 

C 

H 

0.25 

0.14 

0.17 

0.20 

0.10 

0.25 

7 

S 

N 

D 

C 

L 

0.50 

0.43 

0.50 

0.40 

0.40 

0.45 

8 

L 

N 

D 

G 

H 

0.50 

0.43 

0.50 

0.50 

0.30 

0.45 
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Analysis  of  Data 


•  In  a  simple  analysis,  we  account  for  the  impact  of  any 
particular  factor  (e.g.,  instantiation  size  or  age)  by: 

-  Adding  the  responses  for  the  goal  when  a  given  factor  is  set 
"high"; 

-  Subtracting  the  responses  for  the  goal  when  the  same  factor  is 
set  "low";  and, 

-  Dividing  the  result  by  the  number  of  high  (or  low)  settings  (i.e., 
4  in  this  case.) 

•  Let  R(x)  equal  the  response  value  for  instantiation  x. 

•  For  example,  the  impact  of  age  on  PP,  SG  2  (across  all 
instantiations)  is: 

Vi  *[Sum(Responses  when  Age  =  "Existing")  - 
Sum(Responses  when  Age  =  "New")]  = 

Vi  *  [R(2)+R(4)+R(6)+R(8)-R(l)-R(3)-R(5)-R(7)]  =  0.017857 
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Full  Data  Results 


PPSG1 

PPSG2 

PPSG3 

PP  GG  2 

REQM  SG  1 

REQM  GG  2 

Effect  of  size 

0.00 

0.02 

0.00 

0.03 

0.00 

0.01 

Effect  of  age 

0.20 

0.20 

0.25 

0.23 

0.25 

0.21 

Effect  of  location 

-0.58 

-0.55 

-0.58 

-0.53 

-0.50 

-0.51 

Effect  of  customer 

0.00 

-0.05 

0.00 

0.03 

0.00 

-0.01 

Effect  of  complexity 

0.01 

-0.02 

0.00 

0.03 

-0.05 

0.01 

•  Our  conclusion  is  that  instantiation  location  and  age 
have  an  impact  on  process  institutionalization. 

•  All  other  factors  appear  to  have  negligible  impact. 
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Next  Steps 


•  The  analysis  given  here  is  very  elementary. 

-  More  sophisticated  analysis  techniques  may  be  found  at 

http:/ / www.itl.nist. gov/ div898/ handbook/  and  its  references. 

-  Additional  designs,  appropriate  for  many  more  situations,  may 
be  found  at  the  same  location. 


•  Given  the  analysis,  our  next  appraisal  might  be  a 

SCAMPI  B  that  examines  only  two  factors:  location  and 
age. 

-  A  design  using  only  two  factors  is  full  factorial  with  22  =  4 
instantiations. 

-  We  may  wish  to  conduct  the  next  appraisal  with  replication 
against  some  of  the  design  elements,  to  provide  more  insight 
into  institutionalization. 
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What  Have  We  Learned? 


•  DOE  provides  a  technique  to  help  us  choose 
appraisal  representative  samples  in  a  more 
rigorous  manner. 

•  DOE  fits  with  conducting  a  sequence  of  SCAMPI 
appraisals,  leading  to  a  benchmark  SCAMPI  A. 

•  DOE  techniques  may  be  applied  in  a  SCAMPI 
context  with  similar  schedule  duration  to 
traditional  SCAMPIs. 


•  DOE  can  be  a  complex  subject,  but  there  are 
many  software  packages  and  online  and  print 
references  to  make  applying  it  easier. 
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What  Haven't  We  Discussed 


•  DOE  techniques  actually  work  better  for 
planning  SCAMPIs  for  large  OUs  because  there 
are  more  instantiations  available  for  any  given 
design. 

•  Instantiations  that  reflect  some  factor  settings 
may  not  be  available  in  all  OUs  — we  haven't 
covered  how  to  handle  this  situation. 
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DOE  and  Interview  Questions 
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The  Interview  Dilemma 


•  Conducting  interviews  in  an  appraisal  gives  much  the 
same  challenge  as  choosing  a  representative  sample. 

-  There  are  many  questions  to  ask  and  many  people  to  whom  we 
wish  to  ask  them. 


•  How  do  we  choose? 


•  Note:  If  we  have  information  needs,  then  we  will  want 
to  ask  particular  people  specific  questions! 

•  DOE  is  useful  for  general  questions  intended  to  fulfill 
face-to-face  affirmation  coverage  requirements. 


34 


Business 
Transformation 
Institute 


Example:  Designs  for  Interviews 


•  We  can  categorize  personnel  as  "managers",  "engineers",  and 
various  kinds  of  "support". 

•  Each  person  will  also  have  an  instantiation  (possibly  more  than  one) 
associated  with  them. 


•  In  this  case,  the  personnel  categories  and  the  personnel's  binned 
instantiation  provide  the  settings  for  the  factors. 

•  We  choose  the  questions  to  be  asked  of  each  person  based  an 
experimental  design  guiding  us  to  sample  certain  combinations  of 
personnel  categories  and  instantiations. 

•  Due  to  SCAMPI  coverage  requirements,  particularly  for  SCAMPI 
As,  we  will  need  a  fractional  factorial  constrained  design. 

-  Unlike  the  regular  fractional  factorial,  constrained  designs  are  not 
conveniently  available  for  reference. 

-  Our  only  choice  in  this  case  is  to  use  software  that  supports  DOE. 
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Example:  Interview  Design 


•  Note:  this  is  an  example  to  demonstrate  how  the 
technique  might  be  applied,  not  a  real  design. 

-  The  ideas  are  the  same  as  applied  in  choosing  a 
representative  sample,  so  we  will  not  repeat  the 
details! 


•  Suppose  we  have  personnel  categories 
"manager"  and  "engineer". 

•  Suppose  we  have  two  instantiations  to  consider: 
project  1  and  project  2. 
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The  Design 


Question 

Question 

Question 

Question 

1 

2 

3 

4 

Manager, 
Project  1 

Ask 

No 

No 

Ask 

Manager, 
Project  2 

No 

Ask 

Ask 

No 

Engineer, 
Project  1 

No 

Ask 

Ask 

No 

Engineer, 
Project  2 

Ask 

No 

No 

Ask 
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•  DOE  provides  a  powerful  method  for  designing 
reasonable  representative  samples. 

-  DOE  is  of  greatest  benefit  in  dealing  with  large  OUs  with  many 
factors  and  instantiations. 

-  DOE  works  well  in  screening  out  instantiations  that  do  not 
provide  much  "new"  information  through  SCAMPI  Cs  and  Bs. 

•  DOE  provides  a  means  during  an  appraisal  for 
determining  the  general  questions  to  ask  various 
personnel  types  on  different  projects. 

-  Specific  questions  to  answer  information  are  still  directed  as 


usual. 
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Contact  Information 


•  Bob  Moore,  Business  Transformation  Institute,  Inc. 


-  rlmoore@biztransform.net 


•  Will  Hayes,  Software  Engineering  Insitute 

-  wh@sei.cmu.edu 
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Government  Communications 
Systems  Division 


HARRIS 


DoD  Programs 


$1.8B  in  Sales 
8,000  Employees 
ISO  9001 :2000 
CMMI®  Level  3 
SW-CMM®  Level  4 


Civil  Programs 


National  Programs 


is 
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Background 


HARRIS 


Increasing  requirement  for  organizations  to 
demonstrate  compliance  to  CMMI® 

-  Formal  appraisals  (SCAMPISM  A) 

-  Progress  appraisals  (SCAMPISM  B&C) 

Appraisal  preparation: 

-  Time  consuming  &  labor  intensive 

-  Success  depends  on  getting  it  right  the  first  (and 
sometimes  only)  time 

Risk  of  achieving  an  organization’s  process 
maturity  goal  can  be  minimized  through: 

-  Appraisal  experiences,  both  positive  and  negative 

-  Industry  benchmarking 
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Sign  You’re  Ready  (or  Not)  -  #10  hp\nms 

Organization  has  Process  Leadership 

-  Senior  Management  Commitment 

-  Dedicated  Resources 

•  Process  Group 

•  Budget 

•  Tools 

-  Process  Model  selection 

•  CMMI®-SE/SW/IPPD/SS 

•  Staged  or  Continuous  representation 

-  Monitor  and  enforce  Process  Compliance 

•  All  Qualifying  projects 

•  All  Functional  Organizations 
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Improvement  Organization 
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President,  GCSD  Staff 

Steering  Committee  for 
integrated,  division-wide 
process  improvement 

Representatives  from  each 
functional  organization 


Division  Process 
Council 


Working  Arm  of  the  DPC 

Empowered  representatives  from 
each  functional  organization 

Owns  and  maintains  (CCB) 
division-level  process  command 
media  (Integrated  Process 
Manual) 

Monitors  and  enforces  process 
compliance 
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Maturity  Level  Focus 


Process  Areas 


5  Optimizing 


4  Quantitatively 
Managed 


Continuous 

Process 

Improvement 


Quantitative 

Management 


Process 

Standardization 


2  Managed 


Basic 

Project 

Management 


Organizational  Innovation  and  Deployment 
Causal  Analysis  and  Resolution 


Organizational  Process  Performance 
Quantitative  Project  Management 


Organizational  Process  Focus 
Organizational  Process  Definition 

Integrated  Project  Management 

Risk  Management 

Decision  Analysis  and  Resolution 

Requirements  Management 

Project  Planning 

Project  Monitoring  and  Control 

Supplier  Agreement  Management 

Measurement  and  Analysis 

Process  and  Product  Quality  Assurance 

Configuration  Management 


1  Initial 
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Sign  You’re  Ready  (or  Not)  -  #9 


HARRIS 


Organization  has  a  Process  Improvement  Plan 

-  Scope 

•  Establish  how  implemented  in  organization  &  projects 

•  Internal  users  (projects,  managers,  process  group) 

•  External  users  (customers,  appraisal  teams) 

-  Process  Improvement  Organization 

-  Process  Improvement  Objectives 

•  Strategic 

•  Tactical 

•  Measurement 

-  Appraisals 

•  Primary  source  of  candidate  process  improvements 
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Sign  You’re  Ready  (or  Not)  -  #8  hfA.nms 


Organization  has  Integrated  Processes 

-  Organizational  requirements 

•  Process  Model  compliance  (CMMI®) 

-  Integration  and  collaboration  across  functional  organizations 

-  Disciplined  repeatable  processes  with  objective  criteria 

•  Entry/exit  criteria,  inputs,  outputs,  verification,  measures 

-  Planning  each  process,  and  tracking  against  plan 

•  Tailoring  standard  processes  and  assets 

-  Budgets,  schedules,  resources 

-  Managing  established  baselines 

-  Managing  Stakeholder  involvement 

-  Measuring  progress  and  improvement 
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What  is  a  Process? 
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Rll 


'Standard  process 
•Templates,  assets 
•Historical  data 


•Examples 

•Metrics 

•Reporting 


Inputs 


Entry 

Criteria 


Plan 

•Budget 

•Schedule 

•Resources 

•Roles 


Tasks, 
Activities, 
Procedures 


Corrective 

Action 


•Measures 

•CM 


Monitor 


Outputs 


Exit 

Criteria 


V 


raining 


Verification  /  Reviews  /  QA 
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Integrated  Process  Criteria 


HARRIS 


Overview 

A  brief  description  of  the  process  intent 

Entry  Criteria 

Exit  Criteria 

State,  Prerequisites,  Criteria 

State,  Prerequisites,  Criteria 

Inputs 

Outputs 

Required  work  products 

Resulting  work  products 

Required  Activities 

Mandatory  tasks  to  implement  the  process 

Measures 

Process  performance  against  plans 

Organizational  Improvement  Information 

Metrics,  reusable  work  products 

Verification 

Process  compliance  oversight 

Tailoring 

Approved  tailoring,  process  specific 

Implementation  Guidance 

Common  implementation  descriptions 

Supporting  Documentation  and  Assets 

Applicable  GCSD  references. 
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CMMI®  Process  Area  Categories 


Project  Planning 

Project  Monitoring  and  Control 

Supplier  Agreement  Management 

Integrated  Project  Management 
Risk  Management 

Quantitative  Project  Management 


Requirements  Management 

Requirements  Development 
Technical  Solution 
Product  Integration 
Verification 
Validation 


Configuration  Management 
Process  and  Product  Quality 
Assurance 

Measurement  and  Analysis 


Causal  Analysis  and 
Resolution 


Organizational  Process  Focus 
Organizational  Process  Definition 
Organizational  Training 

Organizational  Process 
Performance 

Organizational  Innovation  and 
Deployment 


Maturity  Level  2 

Maturity  Level  3 

Maturity  Level  4 
Maturity  Level  5 
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Integrated  Process  Manual 


HARRIS 


•  Program  Planning  •  Proposal  Development  •  Requirements  Management  •  Process  Improvement 

•  Estimation  •  Requirements  Analysis  *  Risk  Management  *  Training 

•  Program  Monitoring  and  Control  •  System  Architecting/Design  •  Configuration  and  Data  *  Division  Metrics 

•  Supplier  Acquisition  &  •  Design  Management 

Management  •  Code  and  Unit  Test  •  Program  Metrics 

•  Change  Management  •  Fabrication  and  Assembly  *  Decision  Analysis  and 

•  Product  Integration  Resolution 

•  Verification  •  Peer  Review 

•  Validation  •  Design  Review 

•  Production  •  Quality  Assurance 

•  Field  Support  *  Integrated  Logistics  Support 


SEI  Partner 
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Program  Management 


HARRIS 
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Integrated  Process  Manual 


HARRIS 


•  Program  Planning  •  Proposal  Development  •  Requirements  Management  •  Process  Improvement 

•  Estimation  •  Requirements  Analysis  *  Risk  Management  *  Training 

•  Program  Monitoring  and  Control  •  System  Architecting/Design  •  Configuration  and  Data  *  Division  Metrics 

•  Supplier  Acquisition  &  •  Design  Management 

Management  •  Code  and  Unit  Test  •  Program  Metrics 

•  Change  Management  •  Fabrication  and  Assembly  •  Decision  Analysis  and 

•  Product  Integration  Resolution 

•  Verification  •  Peer  Review 

•  Validation  •  Design  Review 

•  Production  •  Quality  Assurance 

•  Field  Support  *  Integrated  Logistics  Support 


SEI  Partner 
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Program  Life-Cycle  Processes  -  1 


IPM 

System 

Architecting/ 

Design 

process 

IPM 

IPM 

IPM 

IPM 

Design 

process 

Fab/Assembly 

IPM 

Proposal 

Requirements 

Analysis 

process 

process 

Product 

Development 

process 

IPM 

Code  and  Unit 
Test  process 

Integration 
process  j 

IPM  Verification  Process 


IPM  Validation  Process 


Program 

Startup 

Review 


<$>  <$>  §).  ($>  $>  ® 


Life-Cycle 

Phase 

Business 

Acquisition 

System 

Requirements 

System 

Design 

Prelim  Design 
Detail  Design 

Fab,  Code, 

Integration 

Verification 

Baseline 

Proposal 

Baseline 

Requirements 

Baseline 

Functional 

Baseline 

•Allocated 

•Design 

Developmental 

Configuration 

Product 

Baseline 

Milestones 

TBR 

SRR 

SDR 

PDR 

TRR 

System  Test 

/  Reviews 

PCR 

CDR 

PC  A,  FCA 

Key 

Products 

Proposal 

Prog  Plans  (P) 
Sys  Arch  (P) 

Prog  Plans 

Requirements 

CONOPS 

Operational 
Threads  /  Use 
Cases 

Sys  Arch 

Sys  Design 
Interface  Defn 

Technical 

Data  Package 

Traceability 

Prelim  Design 
Detail  Design 
Design  docs 

Test  cases  / 
descriptions 

Traceability 

Assembled 

Components 

Component 
test  procs  / 
results 

Integration 
plan  (F) 

Integration 

procedures 

Integration 

results 

Test 

procedures 

Test  results 
Traceability 

Delivered 

systems 
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Program  Life-Cycle  Processes  -  2 


Life-Cycle 

Phase 

Baseline 


Milestones 
/  Reviews 


Key 

Products 


IPM 

Production 

process 

IPM 

Field  Support 
process 

IPM  Verification  process 

IPM  Validation  process 

Other  IPM  Program 
Life-Cycle  processes 
(as  applicable) 

Production 

Field  Support 

Product 

Baseline 

Product  | 

Baseline 

Production 

Readiness 

Review 

Production 

plan 

Delivered 

systems 

As-built 

documents 

Test  results 

Site  Transition  j 
/  Install  Plan 

Revisions  to 
product 
baseline  ! 

Test  results  ■ 

IPM  Production  and  Field  Support 
processes  apply  only  to  the  extent 
required  by  contract 

-  May  be  not  applicable 

-  May  implement  revisions  to  the  baseline 
products 

-  May  involve  other  life  cycle  processes 

•  Requirements,  design, 
implementation 

IPM  Production  Process 

-  Produce  and  deliver  multiple  systems 

IPM  Field  Support  Process 

-  Site  installation 

-  Operations  support 

-  Engineering  services 
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Integrated  Process  Manual 


HARRIS 


•  Program  Planning  •  Proposal  Development  •  Requirements  Management  •  Process  Improvement 

•  Estimation  •  Requirements  Analysis  *  Risk  Management  *  Training 

•  Program  Monitoring  and  Control  •  System  Architecting/Design  •  Configuration  and  Data  •  Division  Metrics 

•  Supplier  Acquisition  &  •  Design  Management 

Management  •  Code  and  Unit  Test  •  Program  Metrics 

•  Change  Management  •  Fabrication  and  Assembly  •  Decision  Analysis  and 

•  Product  Integration  Resolution 

•  Verification  •  Peer  Review 

•  Validation  •  Design  Review 

•  Production  •  Quality  Assurance 

•  Field  Support  *  Integrated  Logistics  Support 


SEI  Partner 
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Program  Support  Processes 


Requirements 

Change 

Requests 


IPM 

Risk 

Management 

process 


•Identify 

•Monitor 

•Manage 


HARRIS 
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Integrated  Process  Manual 


HARRIS 


•  Program  Planning  •  Proposal  Development  •  Requirements  Management  *  Process  Improvement 

•  Estimation  •  Requirements  Analysis  *  Risk  Management  *  Training 

•  Program  Monitoring  and  Control  •  System  Architecting/Design  •  Configuration  and  Data  *  Division  Metrics 

•  Supplier  Acquisition  &  •  Design  Management 

Management  •  Code  and  Unit  Test  •  Program  Metrics 

•  Change  Management  •  Fabrication  and  Assembly  *  Decision  Analysis  and 

•  Product  Integration  Resolution 

•  Verification  •  Peer  Review 

•  Validation  •  Design  Review 

•  Production  •  Quality  Assurance 

•  Field  Support  *  Integrated  Logistics  Support 


SEI  Partner 
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Organizational  Processes 


HARRIS 


Organizational  Processes 


•  Standard  process 

•  Historical  metrics 

•  Process  assets 

•  Trained  staff 


it 


•  Program  metrics 

•  Program  assets 

•  Lessons  learned 
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Sign  You’re  Ready  (or  Not)  -  #7 
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Process  Compliance  is  Audited  &  Monitored 
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Integrated  Compliance  Approach 
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Tailored  Processes 
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Program’s  process  is  tailored  from  “defined  process” 


A 

Accept  IPM  statement  as  written  (no  changes) 

T 

Tailored;  description  of  tailoring  must  be  specified 
(e.g.,  modifications  meeting  intent  of  IPM  statement) 

D 

Deviation;  program  alternative  to  IPM  statement(s),  or  not 
implemented;  Waiver  Approval  required 

N 

Not  applicable;  specify  rationale 

Approved  by  Division  Management 

Establishes  the  approved  baseline  against  which 
process  compliance  audits  are  performed 

Functional  plans  (SEMP,  SDP,  etc.)  are  reviewed  and 
approved  by  cognizant  functional  manager 
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Program  Process  Evidence 


H/urms 


Overview 

A  brief  description  of  the  process  intent 


Entry  Criteria 

State,  Prerequisites,  Criteria 


Exit  Criteria 

State,  Criteria 


Inputs  Outputs 

Needed  work  products,  resources  Resulting  work  products 


Required  Activities 

Mandatory  tasks  to  implement  the  process 

Measures 

Process  performance  against  plans 


Organizational  Improvement  Information 

Metrics,  reusable  work  products 


Verification 

Process  compliance  oversight 


Tailoring 

Approved  tailoring,  process  specific 


Implementation  Guidance 

Common  implementation  descriptions 


Supporting  Documentation  and  Assets 

Applicable  organizational  references 


Program  evidence  needed 
to  demonstrate  I  PM 
process  compliance 
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Process  Compliance 


miznrs 


Integrated  Process  Manual 


Tailoring 

Program  Plans 
Program  process  baseline 
Program  execution 
Compliance  evidence 
QA  verification  — 


Program 

Start-up 


Program  Phase 
Execution 


Program  Appraisals 


Non-compliance  mitigation 


Process 

Compliance 

Monitor 

(PCM) 
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PCM  Project  Workflow 


ASSESSMENT 
STATUS  COLORS 


Process  Compliance  Scores 


HARRIS 


NY 

Not  Yet 

To  be  appraised  at  a  later  date  (i.e.,  the  process  has  not 
yet  been  executed  by  the  process  and  cannot  be 
appraised) 

NA 

Not  Applicable 

Not  applicable  to  the  project  (e.g.,  Code  and  Unit  Test 
Process  is  not  applicable  to  a  production-type  program) 

NS 

Not  Scored 

Pending  an  appraisal 

FI 

Fully 

Implemented 

Direct  artifacts  are  present  and  appropriate 

No  substantial  weaknesses 

LI 

Largely 

Implemented 

Direct  artifacts  are  present  and  appropriate 

One  or  more  substantial  weaknesses 

PI 

Partially 

Implemented 

Direct  artifact  is  absent  or  inadequate 

Substantiated  by  indirect  artifact/affirmation 

One  or  more  substantial  weaknesses 

Nl 

Not 

Implemented 

Any  situation  not  covered  by  the  above 
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Sign  You’re  Ready  (or  Not)  -  #6 


miznrs 


Appraisal  Plan  is  Approved 

-  Purpose 

-  Key  Appraisal  Participant  Information 

-  Appraisal  Scope 

-  Process  Context  Information 

-  Key  Appraisal  Parameters 

-  Planned  Tailoring 

-  Appraisal  Outputs 

-  Appraisal  Constraints 

-  Activities,  Resources  and  Schedule 

-  Milestones  &  Schedule 

-  Risk  Management 

-  Affirmations 
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Sign  You’re  Ready  (or  Not)  -  #5 


miznrs 


Artifacts  are  Managed 

-  Plan 

-  Tools 

•  Configuration  control 

•  Appraisal  with  active  links 

-  Artifact  stability 

•  Broken  links 

•  Moving  data 

•  Aging  data 

•  Version  control 

-  Baselines 

-  Backups 
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Sign  You’re  Ready  (or  Not)  -  #4 


HARRIS 


Artifacts  have  Descriptions  and  Locations 

-  Organizational/project  terminology  and  definitions 

-  Explicit  practice  and/or  artifact  descriptions 

-  Explicit  artifact  titles 

-  Artifact  locations 

•  Online  hyperlinks 

•  Explicit  for  quick  reference 

•  Directories/folders  for  institutionalization 

-  Explicit  Process  Model  tags 

•  CMMI  practice  mapping 

•  Direct,  indirect,  and  affirmation  tagging 
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Process  Compliance  Evidence 


HARRIS 


Direct  Artifacts 

-  Tangible  outputs  resulting  directly 
from  implementation  of  a  practice 

•  e.g.,  plans,  documents,  products 


Required  for: 

•  every  applicable  I  PM  practice 

•  every  applicable  project 


Indirect  Artifacts 

-  Artifacts  that  are  a  side-effect  or 
indicative  of  performing  a  practice 

•  e.g.,  meeting  minutes,  reviews, 
logs,  reports,  metrics 

Affirmations 

-  Oral  or  written  statements 
confirming  or  supporting 
implementation  of  the  practice 

•  e.g,  interviews,  questionnaires 


•  Optional  for  IPM 
compliance  (expected,  but 
not  required). 

•In  formal  CMMI®  appraisals 
(e.g.,  SCAMPISM),  these  are 
required  to  corroborate 
direct  artifacts. 
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Sign  You’re  Ready  (or  Not)  -  #3 


HARRIS 


Evidence  exists  across  Project  Life  Cycle 

-  All  project  phases  intersect  with  one  or  more 
organizational  processes 

-  Direct  and  indirect  or  affirmation  for  each  occurrence 
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Evidence  Collection  across  the 
Project  Life  Cycle 


HARRIS 


Program  Phases 

IPM  Processes 

Business 

Acquisition 

System 

Rqmts 

System 

Design 

Preliminary 

Design 

Detailed 

Design 

Fabrication, 
Code  and 
Integration 

Verification 

Production 

Field 

Support 

Program  Planning 

X 

X 

X 

X 

X 

X 

X 

X 

X 

Estimation 

X 

X 

X 

X 

X 

X 

X 

X 

X 

Program  Monitoring  &  Control 

X 

X 

X 

X 

X 

X 

X 

X 

Supplier  Acquisition  Mgmt 

X 

X 

X 

X 

X 

X 

X 

X 

Change  Management 

X 

X 

X 

X 

X 

X 

X 

X 

X 

Proposal  Development 

X 

Requirements  Analysis 

X 

X 

System  Architecting  &  Design 

X 

X 

X 

Design 

X 

X 

X 

Code  and  Unit  Test 

X 

Fabrication  and  Assembly 

X 

Product  Integration 

X 

Verification 

X 

X 

X 

X 

X 

X 

X 

X 

X 

Validation 

X 

X 

X 

X 

X 

X 

X 

X 

Production 

X 

Field  Support 

X 

Requirements  Management 

X 

X 

X 

X 

X 

X 

X 

X 

X 

Risk  Management 

X 

X 

X 

X 

X 

X 

X 

X 

X 

Configuration  and  Data  Mgmt 

X 

X 

X 

X 

X 

X 

X 

X 

X 

Program  Metrics 

X 

X 

X 

X 

X 

X 

X 

X 

Decision  &  Analysis  Resolution 

X 

X 

X 

X 

X 

X 

X 

X 

X 

Peer  Reviews 

X 

X 

X 

X 

X 

X 

X 

X 

Design  Reviews 

X 

X 

X 

X 

X 

Quality  Assurance 

X 

X 

X 

X 

X 

X 

X 

X 

X 

Integrated  Logistics  Support 

X 

X 

X 

X 

X 

X 

X 

X 

X 

■ 
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Sign  You’re  Ready  (or  Not)  -  #2 


HARRIS 


Evidence  Themes  exist  for  Process  Areas 

-  Within  each  Practice 

•  Indirect  artifacts  compliment  direct  artifacts 

•  Dates  and  versions  are  aligned 

-  Across  practices  within  each  Goal 

•  Artifacts  form  a  common  purpose 

•  Continuity  is  established 

-  Across  goals  within  each  Process  Area 

•  Project  institutionalization  is  understood 

-  Across  projects  with  each  Process  Area 

•  Organizational  institutionalization  is  understood 
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Sign  You’re  Ready  (or  Not)  -  #7  /fAnws 


Evidence  Readiness  Reviews  are  Completed 

-  Pre-appraisal  (SCAMPI)  Readiness  Review 

-  Verification  (99-100%)  vs.  Discovery  (1-0%) 

•  Direct  and  indirect  artifacts  for  every  practice 

•  Relevance  to  practice 

•  Evidence  Themes  exist 

-  Independent  review  is  required 

•  External  appraisers  is  key 

•  Know  vs.  Don’t  Know 


Rno»» 

Don’t 


Don’t 

Know 
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Top  10  Signs  You’re  Ready  (or  Not) 

Evidence  Readiness  Reviews  are  Completed 
Evidence  Themes  exist  for  Process  Areas 
Evidence  exists  across  Project  Life  Cycle 
Artifacts  have  Descriptions  and  Locations 
Artifacts  are  Managed 
Appraisal  Plan  is  Approved 
Process  Compliance  is  Audited  &  Monitored 
Organization  has  Integrated  Processes 
Organization  has  a  Process  Improvement  Plan 
Organization  has  Process  Leadership 


Top  10  Signs  You're  Ready  (or  Not)  For  an  Appraisal 
CMMI®  Technology  Conference  &  User  Group  2005 


communications 71 


SEI  Partner 


Gary  Natwick  -  36 
14-17  November  2005 


Contact  Information 


miznrs 


Gary  Natwick  qnatwick@harris.com 

•  SEI-Authorized  CMMI®  Instructor 

•  SEI-Authorized  SCAMPISM  Lead  Appraiser 

•  SEI-Authorized  SCAMPISM  B&C  Team  Leader 

•  Harris  SEI  Partner  Business  &  Technical  Point  of  Contact 


Harris  Corporation  http://www.harris.com/ 

P.O.  Box  37 

Melbourne,  Florida  32902-0037 

Licensed  by  the  Software  Engineering  Institute  (SEI)  to  provide: 

•  SEI  Introduction  to  CMMI®  courses 

•  SEI  SCAMPISM  Appraisal  Services 


Capability  Maturity  Model  Integration,  CMMI,  and  CMM  are  registered  with  the  U.S.  Patent  and  Trademark  Office. 
SCAMPI  is  a  service  mark  of  Carnegie  Mellon  University. 


Top  10  Signs  You're  Ready  (or  Not)  For  an  Appraisal 
CMMI®  Technology  Conference  &  User  Group  2005 


communications 71 


SEI  Partner 


Gary  Natwick  -  37 
14-17  November  2005 


STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


Welcome 

to  the 


5th  Annual 

CMMI  Technology  Conference 

and 

User  Group 

Sponsored  by 

National  Defense  Industrial  Association 
Systems  Engineering  Division 

With  Technical  CoSponsorship  By 

Software  Engineering  Institute, 

Carnegie  Mellon  University 
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STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


Program  -  Tuesday  Nov  15 


8:30  AM- 12:00  noon  PLENARY  SESSION _ Grand  Mesa  D-E-F 

8:45  AM  -  9:30  AM:  Opening  Keynote 

LTG  Joseph  Yakovac,  USA,  Military  Deputy,  Assistant  Secretary  of  the  Army, 
Acquisition,  Technology  &  Logistics 

9:30  AM  - 10:00  AM  COFFEE  BREAK  Atrium 

10:00  AM  - 12:00  n:  Executive  Panel:  Grand  Mesa  D-E-F 

How  has  CMMI  Improved  Program/Project  Performance  -  -or  has  it??? 

Moderator:  Mr  Mark  Schaeffer,  OUSD(AT&L)  Principal  Deputy,  Defense  Systems  and 
Director,  Systems  Engineering  (also  OSD  Sponsor  of  CMMI) 

Mr.  Dev  Banerjee,  Division  Director,  Systems  &  Flight  Engineering,  Boeing  Integrated 
Defense  Systems 

Mr.  John  Evers,  Raytheon  Processes  Program  Manager,  Raytheon  Common 
Engineering  Process  Program 

Dr.  Arthur  Pyster,  Senior  VP  and  Director  of  Systems  Engineering  &  Integration,  SAIC 
Federal  Systems 

Brig  Gen  Gary  Salisbury,  USAF  (Ret.),  Vice  President  &  Executive  Director,  Busioness 
Development,  Defense  Mission  Systems,  Northrop  Grumman  Corp. 


STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


Luncheon  Speakers 


Tuesday  Grand  Mesa  A/B/C 

Mr.  Clyde  Chittister,  Chief  Operating  Officer,  Software  Engineering 
Institute,  and  Bob  Rassa,  Raytheon,  Industry  Chair,  CMMI  Steering 
Group  and  Industry  sponsor  of  CMMI 

CMMI  -  The  State  of  the  Model 


Wednesday  Grand  Mesa  A/B/C 

Awards  Luncheon  for  Best  Paper  Awards 

Awards  for  Best  Papers  per  Track ,  and  Best  Overall  Paper ,  will  be 
presented  by  Jeff  Dutton,  Jacobs-Sverdrup  Corp 


ThiAwafds  are  sponsored  by  Jacobs  SvaradflUffe&Qipc 

no  speaker 


Conference  ends  at  3:00  PM  after  final  sessions 


STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


1 :30  PM-3:00  PM 


Program  -  Tuesday  Nov  15 


1  CMMI  and  Process  Improvement 

2  Practical  Guidance 

3  CMMI  Appraisals 

4  ROI  &  Benefits  of  CMMI 

5  Acquisition 

6  Transitioning  to  CMMI 

7  CMMI  for  Small  Projects  &  Organizations 
3:00  PM  -  3:30  PM  COFFEE  BREAK 

3:30  PM-5:30  PM 


Grand  Mesa  D/E 
Grand  Mesa  F 
Highlands 
Chasm  Creek 
Mesa  Verde 
Wind  River 
Wind  Star 
Atrium 


All  above  tracks  continue . Track  5  now  High  Maturity 

5:00  -  6:30  PM 


RECEPTION  in  Displays  Area 


Atrium 


STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


Program  -  Wednesday  Nov  16 


8:00  AM-  9:30  AM 

1  CMMI  and  Process  Improvement 

2  Practical  Guidance 

3  CMMI  Appraisals 

4  ROI  &  Benefits  of  CMMI 

5  High  Maturity 

6  Transitioning  to  CMMI 

7  Measurement 

9:30  AM  -  10:30  AM  COFFEE  BREAK 
10:30  AM-1 2:00  Noon 


Grand  Mesa  D/E 
Grand  Mesa  F 
Highlands 
Chasm  Creek 
Mesa  Verde 
Wind  River 
Wind  Star 

Atrium 


All  above  tracks  continue 


STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


Program 


1 :30  PM  -  3:00  PM 

1  CMMI  and  Process  Improvement 

2  Practical  Guidance 

3  CMMI  Appraisals 

4  ROI  &  Benefits  of  CMMI 

5  High  Maturity 

6  CMMI  Extensions 

7  Measurement 

3:00  PM  -  3:30  PM  COFFEE  BREAK 
3:30  PM  -  5:30  PM 


All  above  tracks  continue 


Wednesday  Nov  16 


Grand  Mesa  D-E 
Grand  Mesa  F 
Highlands 
Chasm  Creek 
Mesa  Verde 
Wind  River 
Wind  Star 

Atrium 


Program 


STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


8:00  AM-  9:45  AM 

1  CMMI  and  Process  Improvement 

2  Practical  Guidance 

3  CMMI  Appraisals 

4  ROI  &  Benefits  of  CMMI 

5  High  Maturity 

6  CMMI  Extensions 

7  Systems  Engineering 

9:30  AM  -  10:30  AM  COFFEE  BREAK 
10:15  AM-1 2:00  Noon 


All  above  continue 


Thursday  Nov  17 


Grand  Mesa  D/E 
Grand  Mesa  F 
Highlands 
Chasm  Creek 
Mesa  Verde 
Wind  River 
Wind  Star 

Atrium 


Program 


STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


1 :00  PM-  3:00  PM 

1  CMMI  and  Process  Improvement 

2  Practical  Guidance 

3  CMMI  Appraisals 

4  SCAMPI  B  &  C  Appraisals 

5  High  Maturity 

6  CMMI  Extensions 

7  Systems  Engineering 


Thursday  Nov  17 


Grand  Mesa  D/E 
Grand  Mesa  F 
Highlands 
Chasm  Creek 
Mesa  Verde 
Wind  River 
Wind  Star 


Some  Logistics  Info— 


STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


Message  Number:  (303)  779-1234  (and  ask  for  NDIA  desk) 


Displays  &  Coffee  Breaks  are  in  Atrium. 

Lunches  (Tues,  Wed,  Thursday)  are  in  Grand  Mesa  A/B/C 
at  12:00  noon 


Birds  of  a  Feather  (BOF)  Meetings  are  posted  on  the 
Message  Board  near  the  registration  Area 
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STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


Special  Thanks  To — 


Technical  Program  Chairs: 

Brian  Gallagher,  Software  Engineering  Institute 
Randy  Walters,  Northrop  Grumman 


Session  &  Track  Chairs:  Paul  Croll,  CSC;  Hal  Wilson, 

Northrop  Grumman;  Brian  Gallagher,  SEI;  Dennis  Goldenson,  SEI; 
Geoff  Draper,  Harris  Corp;  Randy  Walters,  Northrop  Grumman;  Jeff 
Dutton,  Jacobs-  Sverdrup;  Rich  Turner,  OSD;  Jerry  Fisher, 
Aerospace  Corp;  Diane  Gibson,  SEI. 


Tutorial  Instructors:  Tim  Kasse,  Kasse  Initiatives  LLC;  Jeff 
Dutton,  Jacobs  Sverdrup;  Rolf  Rietzig,  Cognicence;  Rick  Hefner, 
Northrop  Grumman;  Tim  Olson,  Quality  Improvement  Consultants; 
David  Raffo,  Portland  State  Univ;  Donald  Borcherding,  NexSummit 
LLC;  Mike  Phillips,  SEI;  William  Diebler,  Software  Systems  Quality 
Consulting; 

NDIA  Staff:  Emily  Brown  and  Veronica  Allen 


And  finally — 


STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


Proceedings 

CD-ROM  distributed  at  registration,  contains  all 
presentations  submitted  by  deadline 

Conference  Feedback  Survey 

Fill  out  Survey  Form  (available  at  registration  desk) 
and  give  to  Emily  Brown  or  Veronica  Allen 

We  always  strive  to  improve,  so  we  would  like  your  feedback 

CMMI  Technology  Conference.  Nov  2006 

Call  for  Papers  &  Displays  available  at  registration 
desk 


Implementing  a  Plan  for 
Controlling  Program  ROI  for 
CMMI  ®  Process  Improvement 

J. Perry,  Z.Dennis,  D. Halley,  P.Lenzen, 

S. Madison,  D.Sims 


james.m.perry@baesystems.com 


Objectives 


Use  ‘ROI’  as  a  measure  for 

-  Understanding  and  quantifying  the  benefits/losses 
of  Process  Improvement  (PI)  and  Process  (P) 
performance  for  the  Program 

•  Support  decisions  in  planning  and  performing  PI  and  P 

•  Increase  efficiency  of  Process  Improvement  (PI)  and 
Process  (P) 


Company 


Process  Milestones 


CMM 

|  CMMI 

CMMI 

CMMI 

Level  x 

Updates  to 

Software 

Engineering 

Activity 

Division 

Level  x+1 

1  Level  x-2 

Processes 

PI 

PI 

a 

4 

a 

4 

4 

CMMI 

♦ 

♦ 

4 

CMM 

CM:MI 

CMMI 

CMMI 

Appraisal 

Readiness 

Appraisal 

Appraisal 

F 

/ 

Readiness 

\ppfaisal 

Appraisal 

2003 

2004 

2005 

Toe 

ay 

2006 

Manage  ROI 


Cost  &  Benefit 
Estimates/objective 


Actual  Costs 

-  Management 

-  Process 
Development 

-  Training 

-  Compliance 

-  Appraisals 

-  Execution 


Actual  Benefits 

-  productivity  gain 

-  effort  decrease 

-  cycle  time  decrease 

-  rework  decrease 

-  quality  gain 


*  Data 


*  Control 


Approach 


Benefits  of  the  CMMI 

-  Aggregate  benefits  at  the  process  element  level 
from  CMMI-based  improvements 

•  Define  ‘goodness’  measure  for  a  process  element 

•  Measurements  before,  during,  after  the  improvements 


Process  Improvement  Costs 

(Hours) 

Software 

Engineering  including  Software 

\\ 

\  \ 

2002 

2003 

2004 

2005 

1 


SCI  1 


Sys 

Reqts 


Sw 

Arch. 


Sw 

Reqts 


Taskl 


Design 

► 

Code  & 
Unit  Test 

► 

Sw  Int  & 
Test 

|  Examples  of 
element  level 
process 


Task  2 
Design 


Code  & 
Unit  Test 


Sw  Int  & 
Test 


N 

Design 

► 

Code  & 
Unit  Test 

— ► 

Sw  Int  & 
Test 

SCI  2 


Taskl 


Sw 

Reqts 


Design 

► 

Code  & 
Unit  Test 

► 

Sw  Int  & 
Test 

Task  2 


Design 


Code  & 
Unit  Test 


Sw  Int  & 
Test 


Task  h 

J 

Design 

Code  & 
Unit  Test 

Sw  Int  & 
Test 

Sw 

Qual 

Test 

Sys.  Int 
Test 

W 

W 

1 

r 

Acceptance 

Testing 

Process  Element  Example 


Peer  reviews 
‘Goodness’ 

-  Early  identification  of  defects 

-  Number  of  defects  missed 

-  Time  to  close  defects 

-  Predictability 


Data  Categories  for  Defects 

•  Defects  (Peer  Review) 

•  Change  requests 

-  Developmental  (CUT) 

-  Formal  (Test) 

•  Issues 


Finding  Defects 
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Severity  Occurrence 


Severity  Occurrence 


□  Severity  1 

■  Severity  2 

□  Severity  3 

□  Severity  4 
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Benefits  /  ROI 


Earlier  detection  of  defects 

- 1 .2  defects  per  ksloc  to  .6  defects  per  ksloc  at 
Software  Integration  and  Test 

Time  to  close  defects  decreasing 

Number  of  issues  declining 

ROI  using  cost  of  all  Process  Improvement 
and  only  defect  related  benefits  (ignoring 
all  other  benefits)  =  1 .6 


Summary 


Single  program 

Trend  data  from  2002  to  2005 

-CMM  Level  3  (2002-2003) 

-CMM  Level  4  (2003-2005) 

-  CMMI  Level  5  (2005-) 

Examined  defect  data 

Defect  benefit  /  Total  PI  cost  =  1 .6 
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CMMI 

Current  Appraisal  Synopsis 

Based  on  SCAMPISM  VI  .1  Class  A  appraisals  conducted 
since  April  2002  release  through  August  2005  and  reported 
to  the  SEI  by  September  2005. 

977  appraisals 
878  organizations 
206  participating  companies 
86  reappraised  organizations 
3,686  projects 

59.6%  non-USA  organizations 


*  Organizations  previously  appraised  against  CMMI  VI  .0  and  who  have  not 
reappraised  against  VI  .1  are  not  included  in  this  report. 

Please  visit  http://www.sei.cmu.edu/sema/profile_about.html  for  additional 
information  or  to  find  answers  to  questions  you  may  have  about  this  briefing, 
before  contacting  the  SEI  directly. 
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STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 

Reporting  Organizational  Types 


Commercial/In-house 


64.0% 


Contractor  for 
Military/Government 


31.3% 


Military/Government 

Agency 


0  100  200  300  400  500  600  700 

Number  of  Organizations 


Based  on  878  organizations 


9/30/05 


800 
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CMMI 


Organizations  Type 


Based  on  Primary  Standard  Industrial  Classification  (SIC)  Code 


Mining 

0.3% 


Business  Services 
38.3% 


Engineering  &  Management 
Services 
13.4% 


Retail  Trade 
0.3% 


Wholesale  Trade 
0.6% 


Transportation, 

Communication,  Electric,  Gas 
and  Sanitary  Services 

p  po/ 

^■^/0  Finance,  Insurance  and  Real 

Estate 
6.7% 

Public  Administration 
(Including  Defense) 

11.7% 

Fabricated  Metal  Products 
0.6% 

Primary  Metal  Industries 
0.8% 

Industrial  Machinery  And 
Equipment 
2.0% 

Electronic  &  Other  Electric 
Equipment 
4.2% 

Instruments  And  Related 
Products 
7.0% 


Health  Services 
0.8% 


Transportation  Equipment 
10.1% 


Other  Services 

1.1% 

Based  on  358  organizations  reporting  SIC  code.  For  more  information  visit:  http://www.osha.gov/oshstats/sicser.html  9/30/05 
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Organizational  Size 


Based  on  the  total  number  of  employees  within  the  area  of  the 
organization  that  was  appraised 


1001  to  2000 
6.3% 


2000+ 

3.7% 


501  to  1000 
9.6% 


301  to  500 
10.1% 


201  to  300 
10.9% 


25  or  fewer 
10.3% 


26  to  50 
12.8% 


51  to  75 
9.9% 


76  to  100 
7.9% 


Based  on  861  organizations  reporting  size  data 
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Model  Representations  Selected  for 
Appraisals 
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Disciplines  Selected  for  Appraisals 
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CM  Ml 


STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


Maturity  Profile  by  All  Reporting 
Organizations 
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%  of  Organizations 


CMMI 
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Maturity  Level  of  First  and  Latest 

Appraisal 

60%  T 


50% 


40% 


Not  Given  Initial  Managed  Defined 

■  First  ■  Latest 

Based  on  86  reappraised  organizations  using  their  first  and  latest  appraisal 


Quantitatively 

Managed 


36.0% 


Optimizing 
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STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


. 

CM  Mi 


Countries  Where  Appraisals  Have  Been 
Performed  and  Reported  to  the  SEI 


Argentina 


Australia  Belarus 


Belgium  Brazil 


Canada  Chile 


China 

Germany 

Korea,  Republic  of 
Portugal 

Switzerland 

Vietnam 


Colombia 

Czech  Republic 

Denmark 

Egypt 

Finland 

France 

Hong  Kong  India 

Ireland 

Israel 

Italy 

Japan 

Latvia 

Malaysia 

Mexico 

Netherlands 

New  Zealand 

Philippines 

Russia 

Singapore 

Slovakia 

South  Africa 

Spain 

Sweden 

Taiwan 

Thailand 

Turkey 

Ukraine 

United  Kingdom 

United  States 

Purple  country  name:  new  additions  with  this  reporting  since  Nov.  2004  9/30/05 
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Maturity  Profile  by  All  Reporting 
USA  and  Non-USA  Organizations 
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Number  of  Appraisals  Conducted  by 
Year 


□  SPA  □  CBAIPI  □  SCAMPI  vX  ClassA 
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Number  of  SCAMPI  Version  x.x  Class  A 
Appraisals 

Conducted  by  Year  by  Model  Representation  (where  representation  is 


1999  2000  2001  2002  2003  2004  2005 
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STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 

Number  of  SCAMPI  Version  1.1  Class  A 
Appraisals  Conducted  by  Quarter 

250 

225 
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Appraisal  Results  Summary 


CMMI 


977  appraisals  have  been  reported  since  the  April  2002  SCAMPI  Class  A 
Version  1.1  release. 

Commercial/ln-House  organizations  reporting  appraisals  is  increasing 
more  rapidly  than  other  organizational  categories. 

Government/Military  and  Government/Military  Contractors  reporting 
appraisals  is  increasing  at  a  stable  and  consistent  rate. 

The  highest  percentage  of  Commercial/ln-House  organizations  reporting 
appraisals  is  from  outside  the  USA. 

The  highest  percentage  of  Government/Military  Contractors  reporting 
appraisals  is  from  the  USA. 

Comparing  early  reports  of  the  SW-CMM  maturity  profile  with  early  CMMI 
data  reflects  a  more  mature  CMMI  profile. 

Additional  information  and  charts  will  be  released  as  more  appraisals  are 
reported  and  more  data  is  available  to  support  the  breakdowns. 
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Topics 

Appraisal  Results 
^  Transition  Status 

Product  Suite  VI  .2  Update 
Summary 
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CMMI  Transition  Status  -  9/30/05 


Training 

•  Introduction  to  CMMI  -  38,891  trained 

•  Intermediate  CMMI  - 1 ,738  trained 

•  Introduction  to  CMMI  Instructors  -  372 

•  SCAMPI  Lead  Appraisers  -  577  trained 

Authorized 

•  Introduction  to  CMMI  VI. 1  Instructors  -  290 

•  SCAMPI  VI  .1  Lead  Appraisers  -  398 
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CMMI 

CMMI  Transition  Status  -  9/30/05  2 

Transition  partnering 

Introduction  to  CMMI 

*  190  have  signed 

-  172  are  commercial  offerors  only 

-  14  are  internal-use  only 

-  4  government-use  only 

SCAMPI  Appraiser  Services 

•  213  have  signed 

-  193  commercial  offerors  only 

-  14  are  internal-use  only 

-  6  government-use  only 
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STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


Number  of  Lead  Appraisers 
Authorized  (Cumulative) 


■  Lead  Evaluators 
(SCE) 

■  Lead  Assessors 
(CBA IPI) 

□  Lead  Appraisers 
(SCAMPI) 


1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  YTD 

2005 
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STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


Intro  to  the  CMM  and  CMMI 
Attendees  (Cumulative) 


2005 


^  , 

CMMI 


□  CMM  Intro 

■  CMMI  Intro 

■  CMMI 
Intermediate 
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Number  of  CMMI  Students  Trained 
(Cumulative)  as  of  9/30/05 


30000 


25000 


20000 


15000 


10000 


5000 


0 


1999  2000  2001  2002  2003  2004  YTD 

2005 


■  CMMI 
(Staged) 

■  CMMI 
(Continuous) 

■  CMMI(S&C 
Combined) 


9/30/05 

CMMI  VI  .1  Today  -  The  Current  State  -  page  22 


November  15,  2005 


CMMI  Adoption  Trends:  Website 
Visits  1 

CMMI  web  pages  hits 

•  1.8M/month 

•  Exceeded  60K/day  in  August  2005 

443  organizations  visited  the  CMMI  Website  more  than  200 
times  during  September  2005: 

•  29  Defense  contractor  organizations 

•  12  DoD  organizations 

•  49  Universities 

•  328  Commercial  companies 

•  25  Non-DoD  government  agencies 
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CM  Mi 


CMMI  Adoption  Trends:  Website 
Visits  2 


The  following  were  the 
top  viewed  pages  on 
the  CMMI  Website  in 
September  2005: 

•  CMMI  Main  Page 

•  What  is  CMMI? 

•  CMMI  Models  and 
Modules 

•  Getting  Started  with 
CMMI  Adoption 

•  CMMI  Training, 
Events,  &  Forums 


Average  daily  page  views  per  quarter 
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STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


Performance  Results  Summary  1 


Improvements 

Median 

#  of 
data 
points 

Low 

High 

Cost 

20% 

21 

3% 

87% 

Schedule 

37% 

19 

2% 

90% 

Productivity 

67% 

16 

11% 

255% 

Quality 

50% 

18 

29% 

132% 

Customer 

Satisfaction 

14% 

6 

-4% 

55% 

Return  on 

Investment 

4.8  : 1 

14 

2  : 1 

27.7  : 1 

•  N  =  24,  as  of  9  November  2005 

•  Organizations  with  results  expressed  as  change  over  time 
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CMMI 

Performance  Results  Summary  2 

Credible  quantitative  evidence 

(http://www.sei.cmu.edu/cmmi/results.html) 

•  28  separate  organizations  as  of  November  2005 

•  Major  Website  update  for  CMMI  Technology  Conference  in 
November  2005 

-  Almost  50%  more  data  points  in  the  new  table 

-  Update  includes  a  few  results  where  new  evidence  has 
become  available  since  March  2005 

•  Initial  CMMI  benefits  and  ROI  report,  October  2003 

•  Forthcoming  TR 

•  Frequent  presentations  and  tutorials 

-  CMMI  Technology,  SEPG,  ESEPG,  etc. 

Sources 

•  Conference  presentations 

•  Published  papers 

•  Direct  communication  with  the  SEI 

•  Future  results  from 

-  case  vignettes,  case  studies,  community  benchmarking 
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Summary 
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CMMI 

STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 

CMMI  Version  1.2  Plan 

“Single  book,  single  course”  strategy  begun 

•  VI  .2,  like  the  Addison- Wesley  book,  will  consolidate  both 
staged  and  continuous  representations 

•  Single  course  for  “Intro  to  CMMI”  has  been  created 

-  1st  offering  April  2005 

-  New  instructors  are  being  trained  in  this  single  course 

-  Existing  instructors  have  received  upgrade  training 

-  Staged  and  Continuous  courses  will  be  sunset 
December  2005 

•  Phased  SCAMPI  refinements  will  complement  strategy 

•  Pilot  the  proposed  VI  .2  changes  from  Dec  05  -  Feb  06 

•  Release  VI  .2  Summer  2006 

•  Sunset  VI. 1  after  a  suitable  transition  period 

•  Provide  VI .2  upgrade  training  to  VI. 1  users 
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CMMI 

STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 

Version  1.2  Changes 

•  Eliminate  concept  of  advanced  practices  and 
common  features 

•  Integrate  ISM  into  SAM  at  Level  2 

-  Eliminates  SS  (supplier  sourcing)  option 

•  Recognize,  given  hardware  additions,  that  providing 
separate  development  model  designations  no  longer 
useful 

-  i.e.  “CMMI-SE/SW”,  “CMMI-SW” 

-  “single  book”  approach  (CMMI-DEV+IPPD) 

•  “Not  applicable”  process  areas  (PAs)  for  maturity 
levels  no  longer  an  option  except  for  SAM 
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STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 

Version  1.2  Changes  2 

•  Clarify  material  based  on  1000+  Change 
Requests  (e.g.,  improve  high  maturity  verbiage) 

•  Work  environment  material  added  to  OPD  and 
IPM 

•  Glossary  improved  (e.g.,  higher  level 
management,  bidirectional  traceability, 
subprocess) 

•  Overview  text  improved 

•  IPPD  coverage  consolidated  and  simplified 
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Integrated  Product  and  Process 
Development  (IPPD)  Changes 


IPPD  material  is  being  revised  significantly. 

•  Organization  Environment  for  Integration  PA  removed 
and  material  moved  to  Organizational  Process  Definition 
(OPD)  PA. 

•  Integrated  Teaming  PA  removed  and  material  moved  to 
Integrated  Project  Management  (IPM)  PA. 

•  IPPD  goals  in  the  IPM  PA  have  been  consolidated. 

-  apply  IPPD  Principles 

-  reflects  the  revised  content 

•  Overall  material  condensed  and  revised  to  be  more 
consistent  with  other  PAs. 
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SCAMPI  A  Changes  Being  Considered 
for  VI  .2 

Method  implementation  clarifications 

•  interviews  in  “virtual”  organizations 

•  practice  characterization  rules 

•  organizational  unit  sampling 

Appraisal  Disclosure  Statement  (ADS)  improvements 

•  Add  clarifying  content  to  better  describe  organizational  and 
project  scope  and  coverage 

•  reduce  redundancy  with  other  appraisal  documents 

•  improve  usability  for  sponsor  and  government 

•  require  sponsor’s  signature  on  the  ADS 

Establish  maturity  level  and  capability  level  “shelf  life”  -  3  years 
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STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 

CMMI  “Shelf  Life”  or  expiration  date 

VI. 1  appraisals  will  be  valid  for  three  years  from 
the  time  of  completion,  or  one  year  after  VI. 2 
release,  whichever  is  longer. 

•  VI. 1  appraisal  results  must  be  reported  within 
nine  months  of  appraisal  completion. 

•  VI  .1  appraisals  will  be  accepted  as  valid 
through  December  2007. 

•  Any  appraisal  using  either  the  VI. 1  model  or 
the  VI .1  method  is  considered  a  VI. 1 
appraisal. 

•  The  SEI  will  provide  notification  of  appraisal 
age  at  the  two  year  point. 
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_ _ . 

STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 

Beyond  VI  .2  1 

Improved  architecture  will  allow  post-Vi  .2 
expansion. 

•  Extensions  of  the  life  cycle  (Services, 
Acquisition  &  Outsourcing)  could  expand  use 
of  a  common  organizational  framework: 

-  allows  coverage  of  more  of  the  enterprise  or 
potential  partnering  organizations 

-  adapts  model  features  to  fit  non- 
developmental  efforts  (e.g.,  CMMI  Services, 
CMMI  Acquisition) 

•  Created  concept  of  CMMI  “constellations”  to 
accommodate  the  expansion 
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STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


^  . 

CMML 

Beyond  VI  .2  2 

First  two  constellations,  CMMl  Services  and  CMMl 
Acquisition,  have  been  authorized  by  CMMl  Steering 
Group.  Development  will  be  in  parallel  with  VI  .2  effort; 
publication  sequenced  after  VI. 2  rollout. 


Northrop-Grumman  is  leading  industry  group  for  CMMl 
Services. 

•  Initial  focus  will  be  for  organizations  providing  “DoD 
services”  as  well  as  internal  IT: 

-  e.g.  aircraft  maintenance 

-  Network  Management,  IT  Services 

-  IV&V 
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STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 

Beyond  VI  .2  3 

SEI  is  coordinating  requirements  elicitation  for 

CMMI  Acquisition. 

*  Collecting  government  needs  and 
perspectives  from  both  DoD  and  civil  agencies 

•  Will  build  upon  existing  CMMI  Acquisition 
Module  (CMMI-AM)  plus  an  initial  effort  to 
develop  an  IT  outsourcing  model  (also  based 
on  CMMI-AM) 
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Planned  Sequence  of  Models 
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STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 

Topics 

Appraisal  Results 
Transition  Status 
Product  Suite  VI  .2  Update 
^  Summary 
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STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 

Summary 

Close  to  1000  appraisals  have  been  reported  since  the 
SCAMPI  Class  A  Version  1.1  release. 

CMMI  continues  to  experience  world-wide  adoption. 

•  At  this  time,  approximately  40,000  people  will  have 
taken  the  Introduction  to  CMMI  course 

•  Appraisals  reported  in  13  countries  for  the  first  time 

CMMI  VI. 2  is  nearing  the  piloting  phase  and  will  be 
available  in  the  upcoming  year. 

CMMI  constellations  have  been  commissioned  by  the 
CMMI  Steering  Group. 
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NORTHROP  GRUMMAN 

Defining  the  future 

CMMI: 

Does  it  Help 
Us  Perform? 

CMMI  Technology  Conference, 

Denver,  CO 


November  1 5th,  2005 

Gary  Salisbury 

Vice  President,  Business  Development  and  Sales 

Mission  Systems  Sector,  Defense  Mission  Systems  Division 
Northrop  Grumman  Corporation 


Mission  Systems  Sector 


A  leading  integrator  of  complex, 
mission-enabling  systems 

2004  Sales  -  ~$5.0B 

18,500  employees  in  50  states 
and  in  23  countries 

1500  active  contracts 

Domain  expertise  in  priority, 
high-growth  segments 

Premier  provider  of  mission 
critical  end-to-end  solutions 


Focused  on  program  performance 


NORTHROP  GRUMMAN 
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Does  CMMI  Help  Programs  Perform? _ 

■  CMMI  is  necessary  but,  alone,  is  not  sufficient 

■  Programs  must  also  have 

■  Management  and  Customer  commitment 

■  Reasonable  and  stable  requirements 

■  Adequate  and  trained,  talented  resources 

■  Experienced  organizations  with  domain  knowledge 

■  BUT... 

■  Even  with  all  of  these,  program  organizations  don’t  have 
the  time  to  invent  best  practices 

■  Employment  of  good  process  is  essential  to  success 

■  AND  ... 

■  Using  CMMI  is  the  best  means  we’ve  found  to 
repeatedly  deploy  our  best  processes  and  practices  to 
new  programs 
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What  We  Have  Learned  - 

It's  Not  What  You  Do,  But  When  You  Do  It! 

■  Programs  need  to  apply  key  basic  processes  early 
(even  prior  to  award) 

■  Integrated  Project  Management  (IPM) 

■  Program  Planning  (PP) 

■  Program  Monitoring  and  Control  (PPMC) 

■  Risk  Management  (RSKM) 

■  Requirements  Management  (RM)  and 
Requirements  Development  (RD) 

■  Measurement  and  Analysis 

■  Decision  Analysis  and  Resolution 

■  Good  processes  enhance  good  decision  making 

■  Builds  the  discipline  needed  for  success  under 
‘start  up  stress’ 


We  concentrate  on  putting  proven  processes 
in  place  before  program  kick  off! 
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Mission  Systems  Emphasizes  'Early  Start' 


These  processes  are  mandatory  although  they  can  be 
tailored  or  waived  as  to  the  unique  circumstances  of 
each  particular  program 

■  Process  /  Waiver  Checklist 

■  Every  project  is  required  to  identify  the  processes  that 
it  will  adopt  and  which  it  intends  to  request  to  be 
waived  -  during  the  proposal  phase 

■  Project  kick  off 

■  Every  project  is  required  to  report  on  its  adoption  of 
process  and  progress  towards  execution  in  a  kick-off 
Process  Review 

■  Project  Process  CMMI  Guidance 

■  Every  project  is  provided  with  guidelines  on  which 
process  areas  should  be  in  place  early  and  what 
activities  should  be  included 


CMMI  Technology  Conference  2005 
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Does  It  Work? 


■  Post  award  SCAMPI  Bs 

■  Successfully  demonstrates  that  programs  were 
operating  at  expected  maturity  within  6  months  of 
project  kick  off 

■  Naturally,  some  work  products  that  are  created 
later  in  the  project  life  cycle  won't  be  available 

■  Two  SCAMPI  Bs  in  the  last  few  months 
demonstrate  our  early  process  focus  results  in 
executing  Mission  Systems  processes  from  start  up 

■  Internal  SCAMPI  C  appraisals  ensure  projects  are 
ready  before  we  formally  appraise  them  for 
public  release 


■  Use  of  PIIDs  "on-line"  has  made  electronic 
verification  of  process  deployment  simple  and 
effective  in  a  multi-site  organization 
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Mission  Success  Requires 

Multiple  Approaches 


CMMI  Level  5  for  Software, 
Systems,  and  Services 

ISO  9001  and  AS-9100 
Certification 


NORTHROP  GRUMMAN 
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Early  Starts  Result  in  Proven  Maturity 


■  Mission  Systems  has  made  a  major  investment  in 
process  improvement 


■  We  believe  that  our  emphasis  on  early  start  is  a 
large  contributor  to  these  results 


Organization 

Level  5 

Level  4 

Level  3 

Level  2 

Total 

Northrop  Grumman 

16* 

- 

8 

- 

24 

■  Mission  Systems 

11* 

- 

3 

- 

14 

U.S.  Government 

2 

- 

- 

12 

14 

■  DoD 

2 

- 

- 

7 

9 

Raytheon 

3 

1 

10 

- 

14 

IBM  (International) 

6 

1 

6 

- 

13 

SAIC 

5 

- 

4 

3 

12 

Lockheed  Martin 

2 

3 

6 

- 

11 

Source:  Software  Engineering  Institute  Publicly  Reported  CMMI  Appraisals  website 


*  4  other  appraisals  not  yet  listed 
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DoD  and  CMMI 

November  15,  2005 


Mark  Schaeffer 

Principal  Deputy,  Defense  Systems 
Director,  Systems  Engineering 


CMMI  Vision 


The  initial  vision  for  CMMI  was  to  integrate 
the  competing  maturity  models  and  provide  a 
framework  for  more  consistent  process 

improvement 

•  Cause  integration  of  the  functional  disciplines 
within  organizations  and  across  programs 

•  Increase  systems  engineering  process  maturity 
as  organizations  migrate  from  the  sun-setting 
CMMs  to  CMMI 

Build  on  and  improve  the  significant  work  done  on 

CMM-like  models 


2 


Have  We  Lost  Sight  of  the  Goal? 


•  The  end  goal  of  CMMI  is  to  provide  a  model  for 
continuous  process  improvement  to  achieve: 

-  Reduced  cycle  times 

-  Meet  cost  and  schedule  targets 

-  Improved  quality 

-Combining  Systems  Engineering  and  Software  into  a 
common  model 


When  achieving  a  level  replaces  the  focus  on 
continuous  improvement,  we’ve  lost  sight  of  the  goal 
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How  We  Got  Where  We  Are 


•  CMMI  Sponsors  opted  to  pursue  staged  and  continuous 
models  to  preserve  legacy 

-  SW-CMM,  staged 

-  SECM,  continuous 

•  Acquiring  organizations  do  not  have  full  understanding  of 
how  CMMI  is  intended  to  be  used 

-  What  a  specific  level  at  the  enterprise  level  actually  means  to  an 
acquisition  program 

-  That  the  process  and  people  evaluated  to  obtain  a  level  are  not 
necessarily  applied  to  their  program 

-  Achievement  of  a  specific  level  may  or  may  not  have  meaning  to 
any  given  acquisition  program 
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Negative  Effects  of  “Levels” 


•  Organizations  often  focus  on  maturity  levels  vice  continuous 
improvement 

•  Organizations  are  tempted  to  view  CMMI  Level  “X”  as  an 
“end”  rather  than  a  “means  to  the  end” 

•  Some  organizations  may  stop  at  Level  “X”  because  that  is 
all  that  is  required  or  expected 

•  Level  “X”  companies  often  do  not  perform  at  that  level  on  all 
programs — not  all  programs  are  appraised 

•  Once  an  organization  achieves  a  desired  level,  the  tendency 
is  to  let  the  baseline  erode — can  result  in  reduced  ROI 


DoD  expects  that  if  you  have  achieved  high  maturity, 
the  next  program  will  perform  at  that  maturity 
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CMMI  Workshop,  Sept  7&8,  2005 


•  The  workshop  addressed  several  significant  aspects  of  utilizing 
CMMI  in  the  DoD  and  federal  acquisition  process  that  have  been 
troublesome,  and  developed  recommendations  that  the  CMMI 
Steering  Group,  and  DoD  or  federal  acquisition  agencies  can 
address.  Some  issues  that  were  discussed  include: 

-  Background  on  how  organizations  approach  CMMI  appraisals  and  why 

-  Use  of  Appraisal  Disclosure  Statement  by  acquiring  organizations 

-  Formal  guide  to  CMMI  Usage  for  DoD 

-  Training  for  DoD  Acquisition  Organizations  in  the  use  of  CMMI  for  DoD 

-  Government  lack  of  understanding  of  need  for  mature  SE  content  and 
practice 

-  Specifying  or  requiring  CMMI  in  RFPs 

-  CMMI  Appraisal  expiration  date 
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Workshop  Findings 


•  Programs  execute  at  lower  maturity  levels  than  their  organizations 
have  achieved  and  advertised 

•  Appraisals  use  small  samples — don’t  cover  all  projects 

•  You  can’t  judge  a  program  without  appraising  it 

•  How  can  an  organization’s  level  be  for  “Life”  when  people  and 
processes  change? 

•  High-maturity  practices  are  not  consistently  applied  at  the  project 
level  after  contract  award 

•  Is  the  completeness  of  appraisal  disclosure  statements  adequate? 

•  Low-maturity  acquirers  and  high-maturity  suppliers 


Sound  Familiar? 


7 


Potential  Root  Causes 


•  Lack  of  sufficient  CMMI  guidance  for  acquisition 
professional — what  it  can  do,  what  it  cannot  do, 
applicability  to  source  selections? 

•  Lack  of  tailored  education  and  training  for 
acquisition  professionals — program  managers 
and  contract  officers 

•  Vagueness  with  respect  to  what  an  CMMI  level 
actually  means 
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Way  Ahead 


•  Develop  relevant  guidance  focused  on  multiple 
user  needs 

•  Educate  the  Acquisition  Community 

•  Eliminate  “Level  for  Life” 


Continue  to  improve  the  “application”  of  CMMI 


9 


Raytheon 

Customer  Success  Is  Our  Mission 


Defect  Data  & 
Configuration  Management 


NDIA/CMMI  Conference 
Julie  Schmarje 
Raytheon  Corp. 
November  15,  2005 


CMMI 
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Raytheon 


Topics 

•  Defects  and  the  CMMI 

•  Defect  Collection  and  Reporting  Process  Flow 

•  CM  Role  in  Defect  Collection 

•  Issues  in  CM  Defect  Data  Collection 

•  Summary 


CMMI 
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Raytheon 


Introduction 


•  Raytheon  Space  and  Airborne  Systems  (SAS) 

—  13,000  +  employees 

—  Achieved 

■  CMMI  Level  5,  Software  Engineering,  Sept.  2003,  PSAS  (Texas) 

■  CMMI  Level  3,  Systems  Engineering,  Software  Engineering,  and  Hardware 
Engineering,  August  2005  (California  and  Texas) 

•  -6500  employees 

•  6  Appraisal  Programs 

•  Over  7800  artifacts  collected 

•  Appraisals  Conducted 

•  2  -  Class  C  (February,  July  2005) 

•  3  -  Class  B  (April,  2004,  November  2004,  May  2005) 

•  1  -  SCAMPI  (August  2005) 


CMMI 
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Common  Terms 


Raytheon 


•  Defect:  (n.)  Want  or  absence  of  something  necessary  for 
completeness  or  perfection;  deficiency;  (n.)  Failing;  fault; 
imperfection 

•  The  following  terms  are  used  in  a  generic  manner: 

—  Baseline  -  An  approved  work  product  at  a  specific  revision/version  and 
date.  A  baselined  work  product  is  one  that  is  released  and  controlled  by 
CM. 

—  Change  Request  (CR)  -The  CR  on  programs  could  be  the  IR/CN,  PDM 
Work  Auth,  SCR,  SCN,  STN,  etc. 

—  Configuration  Control  Board  (CCB)  -  The  board  that  reviews  and 
dispositions  CRs  against  baselined  work  products.  The  CCB  on  programs 
that  perform  this  function  could  be  called  any  one  of  a  number  of  names  - 
ERB,  CRB,  SCCB,  CCB,  PRB,  etc. 


CAVMI 


What  is  a  defect  and  why  be  concerned? 
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Defects  and  the  CMMI  (1) 


Raytheon 


•  In  a  CMMI-compliant  Organization  Standard  Process  (OSP), 
defects  are  typically 


-  identified  in  product  reviews  and  audits  as  described  in  the  Verification 
Process  Area  (PA)  (Level  3), 


analyzed  and  used  as  described  in  the  Organizational  Process 
Performance  and  Quantitative  Project  Management  PAs  (Lev< 

and  used  to  perform  root  cause  analysis  to  prevent  similar  def 
future  from  being  introduced  as  described  in  the  Causal  Anal} 
Resolution  PA  (Level  5). 


CMMI 


Identifying,  using,  and  preventing  defects 
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Defects  and  the  CMMI  (2) 


Raytheon 


•For  Engineering,  the  purpose  of  identifying  and  categorizing 
defects  in  selected  work  products  is  to  find  and  remove 
defects  early  in  its  lifecycle. 

*  Studies  have  shown  that  the  later  a  defect  is  identified  in  a 
work  product’s  life  cycle,  the  more  damage  it  causes,  and  the 
more  costly  it  is  to  remove. 


CMMI 


What  is  a  defect  and  why  be  concerned? 
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Defects  and  the  CMMI  (3) 


i 


o 
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Focus  on  continuous  process  improvement 
(the  science  of  process  improvement,  root  cause 
analysis,  ROI,  piloting).  Root  cause  analysis 
performed  on  defect  data  &  other  measures  for 
prevention  of  future  defects. 

Process  measured  and  controlled  quantitatively 
(control  charts,  statistical  analysis,  management  by 
data).  Defect  data  is  analyzed  by  both  projects  & 
the  org.  Corrections  are  made  to  processes. 


Process  characterized  for  the 
organization  and  is  proactive. 

Peer  Reviews  occur  &  defect 
data  reported  at  org  & 
project  level. 

Process  characterized  for 
projects  and  is  often  reactive. 

Measures  defined  &  collected 
by  projects. 

Process  unpredictable,  poorly 
controlled  &  reactive.  Defects 
only  identified  and  removed  in 
later  lifecycle  phases. 


CMMI 


The  emphasis  on  defect  data  increases  with  maturity. 
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Defect  Data  Collection  & 
Reporting  Process  Flow 


RayHieon 


Organization 


Program  Data 
(including  Defect  Data) 


Program  Mgmt 
(Program  “A”) 


Program  Mgmt 
(Program  “n”) 


Peer  Review 
Data 


Customer  Id’d  Defcects 
CR  Data 


Work 
Products 


Engineering  Control  ConfMgmt  Control  Customer 

Control 


CMMI 


Defect  Data  is  collected  from  Peer  Reviews  &  CM 
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Engineering  Control 


RayHieon 


•  During  development,  a  work  product  is  typically  under 
Engineering  Control. 

•  When  ready,  the  work  product  is  Peer  Reviewed,  defects 
identified,  and  categorized  including: 

—  Type  of  defect  found 

—  Phase  where  defect  was  injected 

—  Phase  where  defect  was  found 

•  The  defect  data  is  collected  and  reported  to  Program 
Management. 

•  This  process  is  part  of  the  CMMI  Verification  Process  Area 
(PA). 


CMMI 


Peer  Reviews  provide  defect  data  for  work  products 

under  Engineering  Control. 
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CM  Control  (1) 


Raytheon 


•  Once  the  work  product  has  been  Peer  Reviewed  and  ready  to 
be  controlled,  it  is  given  to  CM  for  baseline  (release). 

•  Once  released,  the  Change  Management  process  begins. 

—  To  change  a  released  work  product,  the  appropriate  Change  Requests  (CRs) 
are  initiated  and  evaluated  by  a  Configuration  Control  Board  (CCB). 

—  The  CCB  analyzes  (or  facilitates  the  analysis  of)  the  change  and,  if 
determined  to  be  a  valid  change,  the  defect  is  identified  and  categorized  on 
the  CR. 


CMMI 


CRs  provide  defect  data  for  work  products  under 

CM  Control. 
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CM  Control  (2) 


Raytheon 


•  CM  collects  and  provides  the  change  and  defect  data  to 
Program  Management. 

•  This  process  is  part  of  the  CMMI  CM  Process  Area  (PA). 


CMMI 


CRs  provide  defect  data  for  work  products  under 

CM  Control. 
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Customer  Control 


RayHieon 


•  Once  work  products  are  delivered,  defects  found  are 
communicated  back  to  the  organization  by  the  customer. 
This  communication  takes  the  form  agreed  upon  by  the 
customer  and  the  program. 

—  A  CR  is  written  documenting  the  defect  found  by  the  customer. 

—  Since  the  work  product  has  been  released  prior  to  customer  delivery,  the 
CR  follows  the  same  Change  Management  process  described  under  CM 
Control. 

•  CM  collects  and  provides  the  customer  change  and  defect 
data  to  Program  Management. 

•  This  process  is  part  of  the  CMMI  CM  Process  Area  (PA). 


CMMI 


CRs  provide  defect  data  found  by  the  customer  on 

delivered  work  products. 
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CMDM  Role  in  Defect  Data  Collection 


RayHieon 


•  During  program  startup,  CM  defines: 

—  the  work  products  to  be  placed  under  CM  Control  and  when  that  control  is 
triggered. 

—  the  Change  Management  process  to  be  used  on  the  program  (including  the 
type  of  CRs  to  use,  the  data  collected  on  them  (fields),  and  the  CR 
lifecycle). 

•  Once  a  work  product  is  given  to  CM  for  release  (baseline), 
CM  ensures  that  the  Change  Management  process  is 
followed  for  any  changes. 

•  When  taking  a  CR  (against  a  released  work  product) 
through  its  lifecycle,  CM  ensures  that  all  fields  on  the  CR  are 
completed,  including  the  defect  data  fields. 

•  CM  coordinates  the  capture  and  analysis  of  defects  found  by 
the  customer  after  delivery. 

•  CM  ensures  that  defect  data  is  captured  on  all  CRs 
CJMHMff  during  the  Change  Management  process. 
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Raytheon 

Issues  in  CM  Defect  Data  Collection  (1) 

•  If  defect  data  is  not  collected  on  CRs  (the  change 
mechanism): 

—  The  only  data  provided  to  the  organization  will  be  from  Peer  Reviews. 

—  Since  the  goal  is  to  find  defects  as  early  as  possible,  any  defects  found  in 
released  work  products  that  escaped  their  phase  (e.g.,  requirements  defects 
found  in  Test)  will  not  be  identified. 


CAVMI 


Defect  data  not  captured  on  CRs  is  a  lost 

opportunity. 
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Issues  in  CMDM  Data  Collection  (2) 


*  Issues  that  impact  the  effective  identification  and  use  of 
defect  data  include: 

—  Programs  that  release  their  work  products  late  in  their  lifecycle: 

■  May  not  be  providing  accurate  defect  data  information  to  the  organization. 

■  May  not  be  documenting  all  changes  that  occur  in  Integration  and  Test  on  CRs; 
therefore,  the  CM  defect  data  will  not  be  accurate. 

-  Method  for  documenting  changes  on  CRs  -  Care  must  be  taken  on  the 

use  of  CRs  for  documenting  changes  and  defect  data. 

■  One  CR  for  each  type  of  defect  identified:  Easier  for  collecting  and  reporting 
defect  data.  Harder  for  overall  change  management  since  more  than  one  work 
product  could  be  affected. 

■  One  CR  for  each  work  product:  Easier  for  change  management,  harder  for 
identifying  and  categorizing  all  associated  defects.  The  CR  would  need  to 
collect  all  instances  of  defects  and  their  categories. 


CAVMI 


To  get  an  accurate  view,  the  appropriate 
CR  process  must  be  used. 
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Impact  of  Defect  Data  Issues 


RayHieon 


•  The  organization  will  not  get  the  opportunity  to  analyze  and 
correct  any  systemic  problems.  Analysis  performed  at  the 
organizational  level  will  be  incomplete  and  conclusions 
reached  may  not  be  valid. 

•  The  organization  will  get  a  false  picture  of  how  effective  the 
program  is  in  early  capture  of  defects. 

•  Levels  4  and  5  could  be  impacted  if  the  organization  isn’t 
provided  with  all  defect  data.  Very  few  defects  will  be 
identified  during  Test  and  Integration  phases  could  raise  a 
red  flag  during  a  CMMI  appraisal. 


CMMI 


Ineffective  CM  defect  data  collection  methods 
impacts  the  organization. 
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EPGs  Role  in  Defining  the  Defect  Data  Process 


Raytheon 


•  The  organization’s  EPG  establishes  the  annual  process 
improvement  objectives  based  on  the  organization’s  goals.  If 
those  goals  include  achieving  a  CMMI  Level  3,  a  proactive 
EPG  will: 

—  evaluate  the  process  needs  for  Levels  3  and  4. 

—  identify  gaps  in  current  processes. 

—  establish  supporting  measurements  and  analysis  processes  through  Level  3 
that  will  easily  pave  the  way  to  Level  4. 

—  ensure  that  all  sources  of  measurement  data  have  a  defined  standard  process 
for  identifying,  categorizing,  and  reporting  (including  defect  data).  This 
would  include  the  CM  Change  Management  process  and  the  collection  of 
defect  data  on  CRs. 


CMMI 


The  EPG  must  ensure  that  mechanisms  are  in  place 
to  collect  and  report  defect  data  from  CRs. 

Copyright  2005  Raytheon  as  unpublished  work.  All  rights  reserved.  Page  1 7 


Summary 


Raytheon 


•  Defect  data  is  identified  in  two  activities: 

—  During  a  Peer  Review  of  a  work  product 

—  During  the  Change  Management  process  of  a  released  work  product. 

•  CM  collects,  maintains,  and  reports  the  defect  data  in  CRs. 

•  Without  the  CM  defect  data,  organizations  will  not  have  all 
the  measurement  data  needed  for: 

—  Evaluating  the  programs  and  organizations  status  in  early  capture  of  defects 

—  Analyzing  and  eliminating  systemic  product  quality  issues 

—  Achieving  CMMI  Levels  4  and  5 


CMMI 
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Managing  Best  Practices 


Adapting  CMMI  Policies  and  Procedures  Used 
in  One  Part  of  an  Organization  to  Another 

Scott  Sherrill 

Georgia  Tech  Research  Institute 


Overview 

•  Questions 

•  About  GTRI 

•  CMM/CMMI  at  GTRI 

•  ELSYS  Laboratory 

•  ITTL  Laboratory 

•  Timeline  for  Implementation  at  ITTL 

•  Prior  to  2004 

•  2004  to  Present 

•  Future 


Georgia  DiragftD'fenft® 
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GTRI  Organizational  Structure 


Administration 

Janice  P.  Rogers 

Business  Operations 

Charles  E.  Brown 

Strategic  Initiatives 

George  B.  Harrison 


Leadership  Council 


Advisory  Council 


Fellows  Council 


Aerospace  Transportation 
&  Advanced  Systems 

James  M.  McMichael 

- 1 — Research  L 
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Tech 
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GTRI’S  National 


\ 


(§)  IPAs 
(§)  Field  Offices 

•  Other  GTRI  Research  Locations 
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Personnel  Statistics 

As  of  October  2005 


Research  Faculty 

552 

Research  Temp/Retired 

94 

Classified  Professional 

29 

Classified  Regular 

238 

Support  Temp/Retired 

126 

Students 

237 

Total 

1,276 
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GTRI  Financial  Statistics 

FY05  Major  Customers 


■  Army 

□  Navy 

■  Air  Force 

□  DoD/Other 

■  Fed/Other 

□  Industry  Fed  Subcontract 

□  Industry 

□  State  &  Local 


Georgia  A  I 
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GTRI  Financial  Statistics 

Historical  Volume 
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CMM/CMMI  at  GTRI 


•  Electronic  Systems  Lab  (ELSYS)  CMM  Level  3 

•  Huntsville  Lab  functioning  at  CMM  Level  4 

•  Information  Technology  and  Telecommunications 
Lab  (ITTL)  has  committed  to  becoming  CMMI  Level  3 

•  ELSYS  and  ITTL  working  to  be  jointly  assessed  on 
CMMI  continuous  model  in  calendar  year  2006 


Comparison  of  ELSYS  and  ITTL 


•  Independently  Managed 

•  Comparable  Size  (ELSYS  slightly  larger) 

•  Both  perform  DOD  centric  work 

•  ELSYS  has  greater  percentage  of  work  from  small  number  of 
customers 

•  ITTL  has  greater  variety  of  customer  types 

•  ELSYS  main  customers  are  requiring  CMM/CMMI 

•  Some  ITTL  customers  also  requiring  CMM/CMMI,  but  many 
others  are  not 


CMM/CMMI  at  ELSYS 


•  Multi  -  Year  Effort 

•  Costs  shared  with  GTRI 

•  Intended  to  be  basis  for  other  labs  becoming  certifed 

•  Certified  CMM  Level  3  in  2003 

•  Processes  defined  by  Engineering  Processes  and  Procedures 
Manual  (EPPM) 

•  Tailored  for  individual  projects 

•  EPPM  being  modified  to  address  CMMI  issues 


CMMI  Issues  at  ITTL 


•  Commitment  to  certify  required  to  work  on  some 
contracts 

•  Certification  necessary  to  bid  on  others 

•  Many  of  our  customers  don’t  care  about  CMMI 

•  Varying  levels  of  motivation  for  certification 


Timeline  for  Implementation  at  ITTL 


•  Prior  to  2004 

•  2004  to  present 

•  Future 


Prior  to  2004 


•  Lab  growing,  recognized  for  doing  good  work 

•  Most  Projects  Well  Managed 

•  Specifics  of  Project  Management  decided  at  Project 
Level 


2004  to  present 

•  January  2004  -  appointed  dedicated  QA  Manager 

•  Told  to  jointly  pursue  CMMI  and  ISO  certification 

•  April  2004  -  QA  Manager  recommends  only 
pursuing  CMMI  certification,  approved  by  lab 
management 

•  April  -  Sep  2004  implementation  plans  developed 

•  September  2004  -  Lab  Director  announces  plan  to 
pursue  certification  to  laboratory 


2004  to  present  (cont) 


•  Sep  2004  -  Initial  implementation  efforts 

•  Joint  Management  Steering  Group  with  ELSYS 

•  Level  of  support  at  project  level  mixed 

•  Strong  support  at  management  level 

•  Projects  had  not  budgeted  for  this  effort  in  current 
projects 


2004  to  Present  (cont) 


•  Projects  expected  to  have 

•  Project  Plan 

•  Written  Requirements 

•  Written  Design 

•  Kept  current 

•  Controlled  Changes 

•  Budget  for  Quality  in  New  Bids 


2004  to  Present  (cont) 


•  EPPM 

•  Large  Document 

•  Evolved  over  5+  years  at  ELSYS 

•  Mostly  applicable  to  ITTL  projects 

•  Too  much  to  implement  all  at  once 

•  Some  may  be  overkill  on  small  projects 

•  Identified  key  processes  for  all  projects 

•  Tailor  as  appropriate  for  individual  projects 


2004  to  Present  (cont) 


•  Inter  lab  relationships 

•  ELSYS  providing  valuable  support  and  guidance 

•  ITTL  usually  accepting  support  and  guidance 

•  MSG  meetings  very  valuable  for  big  picture 

•  Generally  follow  same  processes 

•  Occasionally  we  vary  on  specific  implementation  details 

•  Working  jointly  to  modify  EPPM  to  meet  CMMI  standards 

•  Teamwork  benefits  overall  inter-lab  relationship 


Georgia 

Tech 


Diras'ffl'EtLfl'E® 


2004  to  Present  (cont) 


•  ITTL  QA  Department 

•  Dedicated  QA  Manager 

•  3-4  other  QA  personnel 

•  QA  people  also  do  project  work 

•  Not  on  projects  where  they  do  QA 


2004  to  Present  (cont) 


•  Adopting  existing  EPPM  used  by  another  lab 

•  Very  valuable  -  shown  to  work  for  similar  organization 

•  More  similarities  than  differences  in  work  performed 

•  Some  resistance  due  to  culture  of  autonomy 

•  Much  of  value  of  EPPM  is  not  in  the  EPPM  itself  -  it  is 
in  the  blood,  sweat  and  tears  involved  in  developing  it 

•  Overall,  having  EPPM  to  adopt  is  quite  valuable,  but  it 
doesn’t  remove  the  need  to  learn  from  our  own 
mistakes  -  there  is  value  in  the  journey 


2004  to  Present  (cont) 


•  Support  for  CMMI  processes  within  lab 

•  Initially  quite  varied 

•  Frank  and  open  exchange  of  ideas 

•  Management  support  essential 

•  Many  with  high  level  of  resistance  now  supportive 

•  Still  some  that  are  not 


2004  to  present  (cont) 


•  Northrop  Grumman  external  audit 

•  October  2005 

•  Done  on  JMPS  program  for  their  SAM  process 

•  19  point  checklist 

•  Overall  feedback  very  positive 

•  3  comments 

•  Asked  for  one  supplemental  document 

•  Not  a  real  audit  -  we  have  a  long  way  to  go 


2004  to  present  (cont) 


•  Lessons  (being)  learned 

•  It  will  take  longer  than  you  think 

•  Very  valuable  to  be  helped  by  successful  group 

•  Also  value  in  learning  from  your  own  mistakes 

•  Management  support  and  commitment  essential 

•  People  will  back  you  if  you  can  show  value 


Future 


•  Work  with  ELSYS  to  modify  EPPM 

•  Joint  assessment  2006  (Continuous  Model,  hope  to 
show  at  least  Level  2  on  all  KPAs) 

•  Joint  assessment  2008  (Continuous  Model,  hope  to 
show  at  least  Level  3  on  all  KPAs) 


•  Continuous  Improvement 


•  Questions/Comments??? 

•  scott.sherrill@atri.aatech.edu 


•  (404)894-1190  (until  ~Feb2006) 
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i  Introduction 


There  are  many  process  improvement 
initiatives  that  draw  the  attention  and 
resources  of  senior  management  and  the 
Systems  Engineering  Process  Group  (SEPG) 

Organizations  that  have  a  process  legacy  or 
heritage  in  a  particular  model  or  standard 
have  unintentionally  created  “Rice  Bowls” 


Purpose 


•  This  presentation  will  describe  the  journey  of  a 
Boeing  CMM  (SW)  Level  3  business  unit  that 
was  merged  with  a  Boeing  ISO  9001 :2000 
registered  site  and  how  the  resultant 
organization  shattered  these  Rice  Bowls  on 
the  road  to  CMMI  (SE/SW)  Level  3 


I  The  Initiatives 


Rice  Bowls’ 


AS  91 00 


Setting  the  Stage  @  Boeing 


•  Task:  Integration  of  two  Boeing  business  units: 

-  Business  Unit  A:  CMM  (SW)  Level  3 

-  Business  Unit  B:  ISO  9001 :2000  Registered 


•  Goal:  Unified  process  infrastructure  compliant  to 
CMMI  (SE/SW)  Level  3  and  ISO  9001 :2000 


su 

CMMI 


Iraternationai 
Organization  for 
Standardization 
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Initially  -  Rice  Bowls  Threatened 


Your  CMM 
processes  could 
never  pass  an  ISO 
registration  audit! 


Can’t  we  all 
just  get 
along? 


Oh  yea,  well  your 
ISO  processes 
couldn’t  pass  a  CMM 
appraisal  either! 
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I  Initial  Challenges 


ISO  and  CMMI  have  different  architectures, 
different  languages  and  different  appraisal  methods 

Need  to  set  real  business  goals/objectives  for 
process  improvement  and  try  to  steer  away  from 
goals  like: 

“Achieve  CMMI  Level  X”  or  “Get  ISO  Registered” 

Reason:  Organizations  tend  to  revert  back  to  their 
old  bad  habits  after  achieving  CMMI  or  ISO 
resulting  in  the  return  of  cost/schedule  overruns 
and  poor  quality 


)  Process  Action  Team  (PAT) 


Step  1 :  “Get  over  it!” 

PAT  Mission: 

-  Select  the  “best  practices”  from  the  CMM  (SW)  and 
ISO  process  infrastructures  and  merge  systems 
together 

-  Ensure  compliance  with  both  CMMI  and  ISO 
9001 :2000 

Terminology  Issues:  “I’m  a  CMM  guy  and  this 
ISO  lingo  is  like  a  foreign  language  to  me.” 

Key:  Think  “common  process”  not  CMMI  or  ISO 

Results:  Rice  Bowls  begin  to  shatter  8 


Strengths  of  CMMI  and  ISO 


•  CMMI 


5M 

CMMI 


-  Detailed  Engineering  practices 

-  Comprehensive  Program  Management  practices 

-  Concept  of  increasing  “Maturity  Levels” 


International 
Organization  for 
Standardization 


ISO  9001 :2000 

-  Spotlight  on  Customer  Satisfaction 

-  Focus  on  Control  of  Records 

-  Ensures  process  discipline  across  entire  organization 

-  Annual  Surveillance  Audit 


I  PAT  Focus  -  Our  Customers 


What  do  our  internal  customers  want? 

-Simple  to  use  Process  Asset  Library  (PAL) 

-  One  stop  shopping  for  process,  forms,  work 
instructions,  templates  and  samples 

-  Not  to  deal  with  the  ISO/CMMI  requirements 
-An  “Integrated  Process” 


Compliance  Trace  Matrix 


130  9001:2000 
Description 

CMMI 

CMMI  Description 

ISO 

Implementation 

CMMI 

Implementation 

PA 

Practices 

4 

Quality  Management 
System 

GAPS  in  green 

4.1 

General 

requirements 

U 

Establish  and  implement  the  quality  management  system 

Quality  Manual  &  BMS 

Quality  Assurance  Plan 

Identify  the  processes  that  comprise  the  quality  management  system  and  determine  their  relationships 

Quality  Man.  sect.  4.0 

Organization  Standard 
Process  (OSP) 

Ensure  that  required  resources  are  available 

Quality  Man.  sect.  6.1 

Policy 

Monitor  and  analyze  process  performance  to  ensure  that  the  processes  are  effective 

Quality  Man.  sect.  6.2 

Quality  Assurance  Plan 

Improve  process  performance. 

Quality  Man.  sect.  8.5 

Quality  Assurance  Plan 

Generic 

Goals  & 

Practices 

GP  2.1 

Establish  an  Organizational  Policy 

Quality  Man.  sect.  5.3 

Policy 

Establish  and  maintain  an  organizational  policy  for 
planning  and  performing  the  process. 

Quality  Man.  sect.  5.3 

Policy 

GP  2.2 

Plan  the  Process 

Quality  Man.  sect.  4.0 

Organization  Standard 
Process  (OSP) 

Establish  and  maintain  the  requirements  and 
objectives,  and  plans  for  performing  the  process. 

Quality  Man.  sect.  4.0 

Organization  Standard 
Process  (OSP) 

GP  2.3 

Provide  Resources 

Quality  Man.  sect.  6.1 

Policy 

Provide  adequate  resources  for  performing  the 
process,  developing  the  work  products  and  providing 
the  services  of  the  process. 

Quality  Man.  sect.  6.1 

Policy 
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I  Best  Practice  -  Process  Look  &  Feel 


PI  M2  QMP  -DVTEGKAIION  TEST 


WORK  INSTRUCTION 
PI  2V2  QMI  -  INTEGRATION  TEST 


PURPOSE:  The  □  urease  of  this  process  is  for  the  zes-imn  of  the  'mresrmed"  product  third  ware, 
software  and  interfaces)  received  from  the  PI  201  Product  Iniegrarion  process.  .An  Imegianon  zest 
plan  strategy-  ts  selected  and  defects  are  raptured  arid  corrected  during  the  resting  activity.  When  the 
Integration  Test  :s  complete.  Configuration  MajJfcgsnsnt  i.CMl  takes  control  of  the  product  baseline  in 
preparation  ±n  the  System  and  Beta  Test  phases. 

RESPONSIBILITY 

Proiec:  Manager  (PM)  Tne  PM  is  overall  responsible  foi  die  olanntDg. 

TOOLS 

*  CM  Tool 

*  Defe  z  t  Ti  ickmg  DB 

*  MS  Word 

*  MS  Project 

scheduling,  and  budgeting  associated  with  mtegmnon  resting. 

Integration  Lead:  The  IntezratLon  Lead  will  develoo  and  execute  die 
Integration  Test  Plan  per  the  project  s  Systems  Development  F:an  (SDP). 

The  Integra non  Lead  develops,  the  uitegrarion  strategy,  acquires  the 
necessary'  module  stubs  and  driver s:  oversees  she  integration  zest,  records 
defect  data  and  analyzes  results. 

ConrizuzatLOQ  Management  (CM):  The  CM  lead  is  responsible  for  the 
receiving  die  fund  product  from  die  integration  phase  and  performing  the 
initial  binJd  and  taking  CM  control  of  the  baseline  ar  the  conclusion  of  die 
uitegradcn  phase 

Ufa 

*  Test  Progress 

*  Defect  History' 

ENTRY  CRITERIA 
PITCi  (Product  Integration;:  process 
complex 


EXIT  CRITERIA 

Integration  Ie:i  Report  generated 
Product  imdet  CM  Control 


INPUTS 

OUTPUTS 

SUPPORTING  DOC  UTJENT.ATXON 

*  Assembled  Product 

*  CM  Baseline 

*  System  Specificadon 

(hardware,  software. 

*  Integration  T  es  c  P.  an 

*  Interface  C  c-ntrol  Document  uCD'i 

interfaces) 

*  integration  Te&r  Revert 

*  S  ofrware  De  si  zn  Document  ■  TDD) 

1 ,  Dei'ekp  In  regrati  on  Test  P  la  u  a  nd  Fr  o  redu  t'e? 

a  i  Project  manlier  appoints  an  ImegrStios  L^ia  to  oversee  due  mreeraaoal phase  efforts 
h)  hire  grad  on  Lea  d  develop  3  the  I&segmou  Te::  Phui  sari  procedure :.  The  formality  of 
ne  Integration  Te?t  Plan  and  pr&ceduiet  k  project  dependent,  smaller  projects  may 
develop  an  informal  (internal  vise  only!  Integration  Test  plan  aad  procedures,  hi  this 
care  the  IniegratLDEi  Test  plan  may  be  included  ta  die  overall  project  Tesr  P]an  that 
includes  System  and  Dera  testing.  Larger  projects  -develop  a  formal  (separate : 
Inte.gT3.rion  Test  Plan  and  procedures  that  may  be  a  required  contract  deliverable  item. 

;  1  Di  either  approach  described  m  bl  above,  -he  Ter.  Pi  l  Template  ] scared  on  the 

Prates*  Asset  Library  {PAL.!’  can  be  used 

2.  Receive  Integrated  Product  Components 

5)  hire grari on  Lea d  meets  with  Pro  1  ect  Technic si  L tad  to  prepare  to  receive  the 
m:egra:ed  product  components  for  the  Integration  Test. 

*  Software.  Generally  binaries  from  the  development  platform  will  be  copied  over  to 
the  integration  platform  and  the  Integradcn  Lead  wall  then  control  the  baseline 
tindtigboic:  the  Prodncx  hitegration  Phase. 

*  Hardwire.  The  hardw  are  or  environment  necessary  tor  "die  Iotegranon  less  waL 
be  established  during  the  PI  10 1  Product  lute zr?  ion  pro  z  ess. 

Execute  Imegitiiion  Test 

a.)  IntegiatiDn  Lead  assembles  3  te:t  team  ind  cwen-ees  integrrition  testing  01  the  product 
on  the  integration  platform 

b:  Each  defect  or  act: on  item  is  entered  into  a  de^ct tracking  system  md  assigned  one  01 


developed 
it  a  copy  of 


the  Mlowrag  seventy  codes  by  the  Integra  non  Lead  :NOTE :  the  lo 
Defect  Tracking  Database  cm  be  used  for  tins  process-  see  -he  S! 
this  database): 

Code  1  Most  be  fixed  prior  to  System  Test  m.e.  FacDarf  Ao 
Code  2  Must  be  rixed  prior  to  product  re  lei 
Code  3  Consider  for  future  release 
c  1  The  entire  pro; ect  seam  has  access  10  the  In:egra| 

Win  assign  developers  to  die  correct  defects  in 
problems  assigned  Code  1  or  2.  Corrective  actu 
and  following  lllllt  testing,  the  binanes  ire  agai: 
discussed  in  step  2  above.  Note:  Cannon  should  be  taken  th&?  iotpective  icilods  are 
liimsed  to  defect  rises  and  NOT  new  added  functionality  at  Esi^somt  m  die  lifecycle 
Requirements  or  rimctionality  changes  needio  be  addressed  by  mmagemem  'e.g 
ConfLgiiiatton  Coocrol  Rosrd  ^CCB)). 

d:  The  Integration  Lead  calls  hitegration  test  stants  mee rings  a  1  required  do  discuss 
opened  closed  defects  m  die  database.  This  iterative  process  connnses  riati!  die 
tncegranon  rest  phase  is  successfully  completed. 


T  est  1 


lit.  Die  technical  lead 
feeding  effort  s  on 
:  ^aiented  in  the  database 
d  to  the  integration  platform  is 


Key  Documentation 


Key  Top  Level  Legacy  Documents: 

-  CMM  required  Policy 

-  CMM  required  Organization  Standard  Process  (OSP) 

-  ISO  required  Quality  Manual 

•  Documentation  Merge: 

-  All  3  merged  into  a  new  Quality  Manual 

-  Quality  Manual  format  was  streamlined  to  reflect 
current  organization  policy/practice  and  NOT  mirror 
the  ISO  or  CMMI 


13 


V  Our  Journey  to  CMMI  Level  3 


•  Following  PAT  team’s  efforts,  we: 

-  Chartered  and  staffed  a  new  SEPG 

-  Deployed  updated  process  infrastructure 

-  Deployed  Process  Training  (web  or  instructor-based) 

-  Conducted  CMMI  Level  3  Class  C 

-  Conducted  ISO  Registration  Audit  @  legacy  CMM  site 

-  Conducted  CMMI  Level  3  Class  B 

-  Three  Progress  Reviews  with  External  Lead  Assessor 

-  Conducted  CMMI  Level  3  Class  A  SCAMPI  Appraisal 
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Top  Ten  Lessons  Learned 


1 .  Early  involvement  and  frequent  visits  (Progress  Reviews) 
with  external  CM  Ml  Lead  Assessor 

2.  Open  (constant)  communications  with  ISO  Registrar 

3.  Senior  Management  involvement  and  commitment 

4.  QA  “Mentoring”  before  “Auditing” 

5.  Processes  that  reflect  “Reality”  not  “Theory” 

6.  Proactive/Passionate  SEPG  Members 

7.  Regular  All  Hands  Communications 

8.  Resources  ($$  and  Instructors)  for  Process  Training 

9.  Viable  Process  Improvement  Request  (PIR)  system 

10.  Frequently  Asked  Questions  (FAQ)  email  to  raise 
awareness 
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Sample  FAQ 


The  following  bi-monthly  Frequently  Asked  Question  (FAQ)  is  being  provided  by  the  S&IS 
Mission  Systems  Quality  Department  regarding  our  Quality  Management  System  (QMS) 
process  infrastructure.  To  see  historical  FAQs,  visit  the  Quality.  Management  Homepage 
and  select  the  FAQ  tab. 


Q:  I’ve  just  started  on  a  brand  new  project.  With  regards  to 

establishing  my  required  Engineering  and  Project  Management 
documentation,  where  do  I  begin? 

A:  The  best  place  to  begin  is  with  the  PP  200  Project  Start-up  process. 
This  process  starts  with  a  review  of  our  policy  contained  in  the  S&IS 
Mission  Systems  Quality  Manual  and  then  helps  you  develop  the 
following  key  project  documentation  found  on  our  Templates  Page. 

-  Project  Plan 

-  System  Development  Plan  (SDP) 

-  Risk  Management  Plan 

-  Configuration  Management  Plan 

Additionally,  project  startup  assists  with  establishing  schedules, 
budgets,  tools  and  metrics  on  your  new  project. 
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1  Process  Improvement  Request  (PIR) 


IT . . . 

Process  Improvement  Request  (PIR) 

Date:  09/06/2005 

Submitted  by: 

Project  Name: 

PIR  Title: 

(Check  most  appropriate  choice) 

O  Organization  Artifact  Improvement 
O  Project  Artifact  Improvement 
Hew  Process  Submittal 

Name  of  Affected  Plan,  Process,  Checklist  or  other  Artifact: 

Justification  for  Emergency  or  Priority  classification: 

mprovement  Description: 


Estimated  Benefit  of  Change  (Completed  by  Submitter)! 


Sample  of  Recent  PIRs: 

•  Create  Lessons  Learned  Database 

•  Online  web-based  Peer  Review  Forms 

•  Add  “Operations”  field  for  Defect  data  collection 


(Classification) 

O  Emergency  -  Mission  Critical  or  Safety  Issue  (request  action  w/i  2  days) 
O  Priority  -  Immediate  Project  Heed  (request  action  w/i  10  days) 

Routine  -  All  Others  (request  action  within  20  days) 
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V  1 


What’s  Next? 


ISO  is  helping  to  pave  the  way  for  Integrated 
Product  and  Process  Development  (IPPD) 

Functional  Organizations  (Contracts,  Finance, 
Security,  Fluman  Resources,  etc)  have 
documented  processes  under  ISO  Registration 


IPPD  Effort:  Create  the  “hooks”  with  the  Project 
Management  and  System  Engineering  processes 


CMMI  -  ISO:  The  Bottom  Line 


•  ISO  standard  is  broader  in  scope  and  ensures 
process  discipline  across  the  entire  organization 


•  CMMI  model  provides  greater  detail  and  focus  in 
Engineering  and  Program  Management 

•  Together,  ISO  and  CMMI  models  complement 
each  other  very  well 


International 
Organization  for 
Sta  ndard  ization 


SM 

CMMI 
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I  Q:  Can’t  We  All  Just  Get  Along? 


A:  Yes,  We  Can  and  We  Did! 
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I  Recommended  Reference 


Boris  Mirtafefija  *  Harvey  Slromberg 


Systematic 

Process 

Improvement 

Using 

ISO  9001:2000 


and  CMMI 


Publisher:  Artech  House 
ISBN  1  -58053-487-2 
2003 
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I  Contact  Information 


Dale  R.  Spaulding,  CSQE 

Associate  Technical  Fellow 

The  Boeing  Company 
Space  &  Intelligence  Systems 
15059  Conference  Center  Drive 
Chantilly,  VA  20151-3845 

dale.r.spauldina@boeing.com 

(703)  961-3776 
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Current  Status 


Assessed  CMM  level  3 

Performed  gap  analysis  between  CMM 
and  CMMI 

Updating  processes 
Implementing  the  new  processes 
Not  assessed  under  CMMI 


Georgia 

Tech 


Diras'ffl'EtLfl'E® 


What  is  PPQA? 

Objectively  evaluate  performed  processes, 
work  products,  and  services  against  the 
applicable  process  descriptions,  standards, 
and  procedures 

Identify  and  document  noncompliance  issues 

Provide  feedback  to  project  staff  and  managers 
on  the  results  of  quality  assurance  (QA) 
activities 


•  Ensure  that  noncompliance  issues  are 
addressed 


Small  Project  Assumptions 


*  A  small  project  has  25  people  or  less 

*  Project  team  generally  works  together  on  all 
phases  of  product  development 

*  Must  trade-off  limited  resources 

*  Testers  are  often  the  developers 

*  Need  independent  inspection  at  critical  phases 

*  Quality  engineers  must  have  technical  expertise  to 
add  value  on  a  small  project 


Georgia  DiragftD'fenft® 

I  ®tFTech[ra®D®i]^ 


Very  Small  Projects  (5  or  less) 


•  May  not  have  adequate  funding  to  support  even 
minimal  QA  activities 


Probably  need  more  outside  guidance  and 
independent  reviews  (QA) 


Georgia 

Tech 


Diras'ffl'EtLfl'E® 


Outline 


•  Develop  a  generic  PPQA  plan 

•  Hire  and/or  recruit  Quality  Engineers  highly 
qualified  in  the  product  development  field 

•  Mentor  project  team 

•  Analyze  project  and  product  risks 

•  Build  a  strong  base  for  quality 

•  Add  value  by  reducing  risk 


Georgia  DiragftD'fenft® 

I  ®tFTech[ra®D®i]^ 


Develop  a  Generic  QA  Plan 

•  Developing  a  QA  plan  from  scratch  for  each  project  is  too 
expensive 

•  Many  QA  activities  are  similar  between  projects 

•  Tailoring  a  generic  QA  plan  and  schedule  is  cost-effective, 
and  is  based  on: 

•  Risk 

•  Project  team  experience 

•  Customer  requirements 

•  Project  schedule 

•  Project  deliverables/milestones 


Georgia  A  I 

Tech  Diras'ffl'EtLfl'E® 


QA  Plan  Guideline 

•  Tasks 

•  Start-Up  Tasks 

•  Periodic  Reviews  of  QA  Activities  with 
all  levels  of  organization 

•  Mentor  Project  Team 

•  Support  Customer  QA 

•  Resolve  Disputes 

•  Standards,  Practices,  and  Conventions 

•  Reviews  and  Audits 

•  List  of  required  reviews  (each  phase) 

•  List  of  required  audits  (each  phase,  deliverables) 

•  Peer  review  guidelines 


Georgia  1 

Tech  ^  Diras'ffl'EtLfl'E® 


•  QA  Schedule  Template 


II  Georgia  DirasftD'StLoft® 

M  ®tFTech[ra®D®i]^ 

Hire/Recruit  Qualified  - 
Quality  Engineers 


Technical  and  managerial  experience 


Knowledgeable  in  appropriate  technical  areas 


Should  be  capable  of  doing  “real  work” 

Recognized  by  project  team  for  their 
experience  and  competency 


Able  to  abstract  and  share  information  across 
projects 


Georgia 

Tech 


Diras'ffl'EtLfl'E® 


Georgia  DiragftD'fenft® 
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Mentor  Project  Team 


•  Technical  areas 

•  Management  areas 

•  New  processes 

•  Existing  tools  and 
processes 

•  Attitude 


Georgia  A  I 

Tech  Diras'ffl'EtLfl'E® 


Analyze  Project  and  Product  Risks 


Specific  team  members 

•  Compliant  vs.  noncompliant 

•  Experienced  vs.  inexperienced 
Phases  of  development 

Cost  of  re-work  or  failure 
Familiarity  with  the  subject  area 


Georgia  DiragftD'fenft® 

I  ®tFTech[ra®D®i]^ 


Build  a  Strong  Base  for  Quality 


Leverage  “star  players” 

•  spread  across  project  teams 

•  use  to  develop  processes 

Praise  “star  players”  and  reward  them 
to  the  extent  that  you  are  capable 


•  Modify  processes  to  the  organization's 
best-in-class 


•  Create  an  environment  where  process 
compliance  is  institutionalized 


Georgia  A  I 

Tech  Diras'ffl'EtLfl'E® 


Georgia  DiragftD'fenft® 

I  ®tFTech[ra®D®i]^ 


Add  Value  by  Reducing  Risk 


*  Prioritize  organizational  QA  activities  based  on 
project/product  risk 


•  Communicate  status  to  all  levels  of  the 
organization,  as  appropriate 

•  Share  lessons  learned  for  all  projects 

•  Assist  the  project  team  in  developing  and 
implementing  risk  mitigation  strategies 

•  Act  as  “the  conscience”  of  the  project  team 


Georgia  A  I 

Tech  Diras'ffl'EtLfl'E® 
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AUTHORS:  JEAN  SWANK,  JEANNE  BALSAM,  LEE  SHEINER,  AND  MARK  PELLEGRINI 


INTRODUCTION 


Georgia  Tech  Research  Institute  (GTRI)  is  the  nonprofit  applied  research  arm  of  the  Georgia 
Institute  of  Technology  in  Atlanta,  Georgia.  The  Electronic  Systems  Laboratory  (ELSYS)  of  the 
GTRI  achieved  a  CMM  Level  3  rating  in  June  of  2003.  ELSYS  employs  approximately  150 
engineers  and  scientists  working  predominately  on  DoD  related  competitively  bid  contracts.  Over 
the  last  30  years,  ELSYS  researchers  have  established  national  reputations  in  areas  such  as: 
monopulse  countermeasures,  advanced  radar  warning  receiver  design,  survivability,  simulation 
models  and  analysis,  and  Electronic  Counter  Measures  (ECM)  technique  development. 
GTRI/ELSYS  core  competencies  include  software  and  systems  engineering  for  electronic  warfare 
and  avionics  systems,  reliability  and  maintainability  upgrades,  technology  insertion,  obsolescence 
programs,  threat  analysis,  and  mission  critical  software.  Many  of  these  projects  are  staffed  by  fewer 
than  ten  developers;  some  projects  have  only  one  or  two. 

ELSYS  has  transitioned  the  CMM  Software  Quality  Assurance  process  to  the  CMMI  Process  and 
Product  Quality  Assurance  (PPQA).  This  presentation  will  share  lessons  learned  while  transitioning 
to  a  fully  compliant  systems  engineering  PPQA  function.  The  reasons  for  the  following  statements 
will  be  discussed: 

1)  Develop  a  generic  QA  plan  and  schedule  that  can  be  easily  tailored  for  specific 
project/product  needs. 

2)  Hire  and/ or  recruit  Quality  Engineers  that  have  enough  experience  that  management 
respects  their  recommendations.  These  people  can  supplement  technical  and 
managerial  expertise  of  the  project  team  which  visibly  adds  value  to  the  development 
effort. 

3)  Have  Quality  Engineers  act  as  mentors  to  the  project  team. 

4)  Analyze  project  and  product  risks  to  determine  the  most  cost  effective  PPQA 
strategy. 

5)  Leverage  project  team  process  maturity  for  reduction  of  PPQA  tasks  and  for  process 
improvement.  Praise  process  innovators  and  reward  them  to  the  extent  that  you  are 
capable. 

6)  Concentrate  PPQA  on  projects  or  product  development  efforts  that  have  high  risks; 
for  example  safety  or  technology. 

This  paper  describes  a  method  to  implement  effective  PPQA  in  small  organizations  or  small 
projects  in  order  to  produce  best  in  class  products  with  limited  resources. 


SMALL  ORGANIZATIONS  AND  PROJECTS 


In  the  context  of  this  paper,  “small  organizations”  refers  to  organizations  of  150  or  fewer  people 
with  projects  ranging  in  size  from  one  to  twenty-five  people.  The  resources  a  small  organization  is 
able  to  commit  to  PPQA  are  generally  severely  limited.  In  these  organizations,  a  Quality  Engineer 
typically  cannot  be  assigned  full-time  to  a  single  project. 
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A  small  organization  may  lack  development  phase  specialization  in  its  members.  The  same  people 
who  write  the  requirements  may  also  be  the  testers.  The  designers  might  be  the  implementers.  In 
very  small  organizations  it  may  be  the  same  people  performing  all  of  the  development  activities  from 
start  to  finish. 

This  raises  an  interesting  dilemma.  What  is  more  important,  minimal  quality  assurance  distributed 
equally  on  all  projects  or  more  comprehensive  quality  assurance  on  some,  with  others  receiving  very 
little?  When  different  teams  of  people  write  requirements,  design,  implement,  and  test  a  product, 
there  is  a  sort  of  “built-in”  protection  system.  If  the  group  writing  the  requirements  does  a  poor 
job,  the  designers  will,  hopefully,  complain  that  they  have  been  given  requirements  that  are  too 
vague,  or  the  testers  will  complain  that  they  are  un-testable.  Likewise,  if  the  coders  are  given  a  poor 
design,  they  may  alert  management  that  there  is  a  problem.  However,  when  the  same  people  are 
doing  all  of  the  development  from  start  to  finish,  they  can  walk  down  a  primrose  path,  not  realizing 
they  have  a  disaster  in  the  making.  It  is  these  projects  that  need  PPQA  the  most,  yet  they  can  afford 
it  the  least.  This  is  the  challenge  of  making  PPQA  work  on  small  projects. 

Large  organizations  may  have  projects  with  multiple  Quality  Engineers  assigned  full-time  to  them, 
but  they  can  still  have  some  projects  that  are  quite  small,  with  limited  resources  and  role-sharing.  A 
large  organization  may  experience  minimal  impact  from  the  failure  on  one  of  its  smaller  projects, 
whereas  a  small  organization  may  well  experience  severe  consequences. 


PROCESS  AND  PRODUCT  QUALITY  ASSURANCE 


According  to  the  Software  Engineering  Institute,  “The  purpose  of  Process  and  Product  Quality 
Assurance  is  to  provide  staff  and  management  with  objective  insight  into  processes  and  associated 
work  products.1”  For  small  projects  there  is  one  key  word  in  this  phrase  unique  to  those  projects: 
objective.  Generally,  everyone  on  a  very  small  project  has  fairly  good  insight  to  what  is  happening 
on  the  project;  what’s  missing  is  an  objective  set  of  eyes.  On  large  projects  there  is  inherently  some 
objective  oversight  from  other  team  members.  Regardless  of  whether  the  project  is  large  or  small, 
management  external  to  the  project  should  be  kept  objectively  informed  of  the  technical  and  process 
status  of  the  project. 


PLANNING 


Through  years  of  experience,  GTRI  has  determined  that  having  a  generic  Quality  Assurance  (QA) 
Plan  for  the  organization  is  the  most  effective  both  in  cost  and  functionality.  Additionally,  a  generic 
schedule  that  includes  all  tasks  required  by  the  organization’s  standard  process  is  used  as  the  starting 
point  for  each  project.  A  database  is  used  to  track  these  schedules  and  any  other  supplemental 
material  that  is  project  or  product  specific.  Over  time  the  organization  should  develop  a  library  of 
generic  schedules  for  each  product  development  type.  The  generic  QA  Plan  serves  approximately 
seventy-percent  of  the  projects  without  revision.  The  QA  Plan  may  be  supplemented  to  address 
specific  tailored  processes,  risks,  and  mitigation  strategies. 


1  Mary  Beth  Chrissis,  Mike  Konrad,  and  Sandy  Shrum,  CMMI  (Addison  Wesley,  2003),  429. 
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The  guidelines  shown  in  Table  1  are  provided  as  an  example  for  developing  a  generic  QA  Plan  for 
the  organization.  In  general  the  QA  plan  needs  to  be  consistent  with  plans  developed  for  other 
purposes.  It  should  include  the  introductory  sections  including:  Identification,  Scope,  Document 
Overview,  Referenced  Documents,  and  Organizational  Structure.  Additionally,  it  should  have 
standard  tasks  that  parallel  those  of  the  organization’s  standard  process. 


Table  1 


Section  Name 

Description 

>  Tasks 

This  section  provides  an  outline  with  detailed  expectations  for 
each  type  of  audit  and  review  that  QA  will  provide  oversight  and 
support. 

o  Perform  Start-Up  Tasks 

These  tasks  are  normally  executed  a  single  time,  but  may  be 
repeated  for  major  contractual  changes  or  for  incremental 
developments  where  this  level  of  coordination/re-planning  is 
necessary. 

■  Attend  Project  Initiation  Meeting 

■  Generate  QA  Plans  and  Schedules 

■  Prepare  for  and  Attend  the  Plan  Review 

Meeting 

■  Prepare  Orientation  for  and  Attend  Project 
Kick-off  Meeting 

■  Attend  Customer  Meetings 

o  Conduct  Periodic  Reviews  of  QA  Activities 

Explicitly  define  what  reviews  of  QA  activities  are  required.  Define 
what  type  of  data  is  shared  at  each  level  and  the  minimum 
frequency  of  communication. 

■  QA  Manager 

■  Lab  Director 

■  Senior  Management 

■  Project  Director 

■  Project  Team 

o  Mentor  Project  Team  in  Organizational  Process 
Activities 

Define  general  mentoring  activities  that  Quality  Engineers  will 
conduct  during  the  life  of  the  project,  including  the  value  of  those 
activities. 

o  Support  Customer  Quality  Management  System 

Define  minimum  types  of  support  that  will  be  provided  by  the 

Quality  Engineer  to  the  customer. 

o  Resolving  Disputes 

Define  methods  for  resolving  disputes  between  the  Quality 

Engineer  and  the  project  team. 

>  Documentation 

Define  where  additional  documentation  associated  with  QA 
activities  will  be  stored  (may  be  by  reference). 

>  Standards,  Practices,  and  Conventions 

Define  where  the  official  organizational  standards,  practices, 
policies,  guidelines,  and  conventions  are  located. 

o  Tailoring  of  Standard  Process 

Reference  tailoring  practices  for  the  organization’s  standard 
process. 

o  Monitoring  Compliance 

Define  how  compliance  will  be  monitored  and  how  deviations  will 
be  processed. 

>  Reviews  and  Audits 

Define  who  shall  conduct  reviews  and  audits  and  how  the 
respective  processes  and  standards  will  be  used  in  these  reviews. 

o  Technical  Reviews 

Reference  applicable  procedures  and  standards  for  conduct  of 
technical  reviews.  Include  the  level  of  detail  necessary  for  each 
type  of  review  and  audit  to  be  conducted.  Define  how  data 
collected  during  the  reviews  will  be  analyzed  and  reported. 

■  Conduct  Periodic  Reviews  of  Project  Activities 

•  Configuration  Management 

•  Software  Product  Engineering  Process 

•  Hardware  Product  Engineering  Process 

■  Peer  Reviews 

■  Technical  Audits 

•  Collect  Measurement  Data  and  Document 
Deviations 

•  Analyze  Data  and  Report  Results 

■  Managerial  Reviews 

o  Configuration  Management 

Detail  specific  processes  for  configuration  management  audits. 

o  Problem  reporting  and  corrective  actions 

Define  how  problems  are  reported  and  corrective  actions  are 
tracked  to  closure. 

o  QA  Document  Identification  Conventions 

Define  document  identification  conventions  for  QA  artifacts. 
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QUALITY  ENGINEERS 


To  add  the  greatest  value  it  is  essential  that  quality  engineers  be  well-qualified  in  both  managerial  and 
technical  areas.  In  GTRI,  Quality  Engineers  are  required  to  have  technical  degrees  in  either 
computer  science  or  engineering,  with  practical  experience  in  product  development  including  project 
management.  In  additional  to  being  familiar  with  the  organization’s  defined  engineering  processes, 
they  also  need  to  understand  project  planning  and  risk  management.  They  are  capable  of 
performing  technical  and  managerial  work,  although  they  are  assigned  to  the  project  as  an 
organizationally  independent  monitor. 

Having  the  ability  to  do  the  actual  work  makes  a  Quality  Engineer  far  more  valuable  than  a  “box- 
checker.”  Certainly,  checking-the-boxes  along  the  development  path  is  better  than  having  no 
independent  verification  at  all,  but  some  problems  do  not  lend  themselves  to  discovery  simply  by 
seeing  if  the  proper  box  has  been  checked.  Technically  trained  Quality  Engineers  are  more  likely  to 
detect  trouble  on  a  project  (and  more  quickly)  than  ones  who  do  not  understand  the  technical 
aspects  of  the  product  whose  development  processes  they  are  monitoring. 

Highly  qualified  Quality  Engineers  bolster  the  capabilities  of  the  project  team  because  they  add  an 
independent  set  of  eyes  to  the  project  team.  They  attend  meetings,  review  project  documentation, 
and  are  aware  of  what  the  team  is  doing.  Not  only  can  they  help  spot  problems,  they  can  provide 
suggestions  and  advice  that  are  valued,  as  they  are  recognized  by  the  team  as  being  qualified  to  do 
so.  Quality  Engineers  who  participate  in  multiple  projects  provide  another  valuable  service  —  they 
can  share  technical  information  between  these  projects  in  a  way  that  someone  who  is  simply 
checking-the-boxes  never  could.  This  can  help  avoid  conflicts  between  products  that  share 
requirements  or  resources.  Often  the  Quality  Engineer  can  share  solutions  developed  on  one 
project  with  a  different  project  team  that  is  having  similar  problems. 


MENTORING 


Classroom  training  alone  is  seldom  enough  to  provide  a  practice  capability.  In  small  organizations 
typically  it  is  expected  that  employees  will  learn  through  self-study  and  informal  mentoring  from 
other  project  team  members.  The  Quality  Engineer  is  in  a  good  position  to  identify  developers  who 
are  in  need  of  mentoring  in  order  to  develop  a  practice  capability  with  the  organization’s  standards 
and  processes. 


ASSESSING  QUALITY  RISKS 


In  order  to  properly  apply  scarce  quality  assurance  resources  to  projects,  it  is  first  necessary  to 
identify  the  highest  areas  of  risk.  A  number  of  factors  need  to  be  considered. 

Personnel  —  Knowledge  of  the  capabilities  and  work  habits  of  the  people  on  a  project  team  can  be 
valuable  in  deciding  where  to  allocate  resources.  If  the  team  members  are  known  to  generally 
conform  to  the  organization’s  defined  processes  —  with  everything  else  being  equal  —  it  would  be 
more  effective  to  allocate  quality  assurance  resources  to  other  project  teams  that  are  known  to  be 
less  process  compliant  or  technically  challenged.  Quite  simply,  it  makes  sense  to  spend  time  looking 
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for  process  violations  among  people  who  have  a  history  of  violating  the  organization’s  processes. 
Another  personnel  factor  is  the  level  of  technical  experience  of  the  team.  Inexperienced  developers 
would  ordinarily  warrant  closer  inspection  than  those  who  are  veterans.  If  the  Quality  Engineers  are 
well-trained  and  capable  of  doing  technical  work,  as  the  authors  contend  they  should  be,  they  can 
periodically  sample  the  work  and  sound  a  warning  if  there  appears  to  be  a  problem.  In  any  case, 
sufficient  and  appropriate  peer  reviews  of  an  inexperienced  developers’  work  should  be  conducted; 
proper  quality  assurance  verifies  that  these  reviews  are  being  scheduled  and  completed. 

Development  Phases  —  Ideally,  a  defined  process  would  be  followed  throughout  the  entire  lifecycle 
of  every  product,  including  a  continuous  verification  by  quality  assurance  that  the  product  is  being 
built  correctly.  However,  in  the  absence  of  enough  resources  to  verify  continuously,  there  are 
certain  key  phases  of  development  where  the  quality  of  the  output  needs  to  be  verified;  for  example, 
the  requirements,  design,  development,  and  testing  phases.  Unquestionably  it  would  be  better  to 
have  requirements  written  correctly  from  the  start,  but  if  there  is  a  problem  with  them,  it  is  better  to 
detect  and  correct  the  problem  before  they  are  used  as  the  foundation  for  design  rather  than 
afterwards.  Likewise,  a  poor  design  should  be  identified  and  corrected  before  it  is  implemented.  If 
the  quality  assurance  budget  only  permits  limited  involvement  of  a  Quality  Engineer  in  the  product’s 
development,  it  is  better  to  schedule  the  time  at  the  critical  phases,  rather  than  concentrating  in  a 
single  phase.  A  set  of  rock-solid  requirements  is  a  good  start,  but  if  the  entire  quality  assurance 
budget  was  spent  on  their  development  and  the  project  goes  astray  during  design,  this  is  not  a  good 
trade-off. 

Cost  of  Failure  —  Sometimes  if  a  product  fails,  the  cost  of  failure  can  exceed  the  money  spent  on 
developing  it.  For  example,  it  could  be  a  key  component  of  some  other  much  larger  product  or 
system  whose  success  is  dependent  upon  the  smaller  one.  Loss  of  reputation  or  team  morale  is  also 
an  important  consideration.  But  sometimes  a  small  product  is  just  a  small  product,  and  if  it  fails  it 
doesn’t  have  dire  consequences  for  the  organization.  If  two  projects  have  an  equal  chance  of  having 
problems,  but  one  has  far  greater  consequences  to  the  organization  if  it  fails,  it  makes  sense  to  put 
more  resources  on  the  one  that  is  more  important. 

Familiarity  with  the  Subject  Area  -  If  the  product  being  developed  uses  new  technologies,  is 
planned  for  deployment  in  unfamiliar  environments  or  has  problems  that  the  organization  has  never 
faced  before,  it  is  probably  a  good  candidate  for  more  quality  assurance  resources  than  one  without 
these  challenges.  If  it  is  a  new  product  that  is  very  similar  in  function  and  scope  to  an  earlier 
product,  it  will  pose  less  risk  than  an  unfamiliar  one.  However,  the  experience  of  the  project  team 
needs  to  be  considered.  Even  if  the  product  is  very  similar  to  other  ones  that  the  organization  has 
created,  if  the  project  team  has  no  direct  experience  with  the  similar  products  the  risk  may  still  be 
high. 


ADDRESSING  PROJECT  RISKS 


The  job  of  the  quality  engineer  is  to  make  sure  that  project  risks  are  assessed  and  documented. 
Additionally,  quality  risks  should  be  factored  in  when  assigning  resources.  In  ELSYS,  the  project  is 
responsible  for  funding  quality  assurance.  If  the  project  lacks  sufficient  funding,  then  corporate 
resources  may  be  used  to  insure  sufficient  oversight  is  given  to  the  project.  After  quality  and 
technical  risks  have  been  assessed,  the  risk  mitigation  plan  should  be  implemented  and  monitored. 
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INSTITUTIONALIZING  PROCESSES 


In  a  perfect  world  there  would  be  no  need  for  quality  assurance  activities;  everyone  would  be 
qualified  to  do  their  job,  they  would  do  it  perfectly  every  time,  and  everyone  would  follow  the 
organization’s  procedures  for  developing  products.  The  world,  unfortunately,  isn’t  perfect.  Neither 
is  it  completely  imperfect,  where  every  developer  needs  a  full-time  Quality  Engineer  sitting  next  to 
them  watching  everything  they  do.  The  reality  is  somewhere  in  between. 

The  most  effective  way  to  utilize  good  processes  to  create  outstanding  products  is  to  create  an 
environment  where  the  project  teams  want  to  follow  these  processes,  rather  than  do  it  because  they 
are  forced  to  do  so.  Thus,  process  improvement  for  product  development  is  more  effective  when  it 
comes  from  the  bottom  up,  rather  than  from  the  top  down.  The  people  doing  the  work  suffer  the 
consequences  of  their  own  mistakes,  and  they  can  identify  the  ones  that  could  have  been  avoided 
through  better  processes.  The  most  motivated  of  these  people  will  take  it  upon  themselves  to  tailor 
or  to  extend  the  organization’s  processes  to  meet  their  needs.  The  Quality  Engineer  is 
management’s  representative  “in  the  trenches,”  and  can  identify  process  improvements  that  should 
be  more  generally  distributed.  Some  of  these  improvements  will  be  generally  applicable  within  the 
organization  and  should  be  incorporated  as  changes  to  the  defined  processes. 

The  organization  needs  to  identify  “star  players”  who  utilize  existing  processes  and  work  to  improve 
those  processes,  and  those  individuals  who  may  not  necessarily  improve  processes,  but  comply  with 
them.  These  people  should  be  praised,  rewarded,  and  encouraged  to  continue  doing  so.  They 
become  role  models  for  the  other  developers,  encouraging  them  to  be  compliant  and  innovative  as 
well.  When  the  project  team  is  voluntarily  and  enthusiastically  following  the  organization’s 
processes  institutionalization  occurs  and  the  need  for  quality  assurance  is  diminished,  reducing  the 
need  for  those  scare  resources. 


PRACTICAL  CONSIDERATIONS 


Responsibility  for  project  outcome  rests  on  the  project  manager’s  shoulders  and  those  of  senior 
management.  As  such,  quality  engineers  are  responsible  for  bringing  concerns  to  the  attention  of 
the  project  managers  and  if  necessary,  senior  management.  The  quality  engineer  acts  as  the 
conscience  of  the  organization,  not  the  police.  Senior  management  must  support  the  quality 
engineer  with  a  concrete  stance  on  processes  and  policies.  In  order  for  senior  management  to  give 
that  support  they  must  respect  the  decisions  of  the  quality  engineers.  Tailoring  of  processes  should 
be  allowed  when  it  makes  sense.  Variance  should  be  approved  when  necessary.  Quality  of  the 
product  should  always  be  the  guiding  value,  not  who’s  in  charge.  Thus,  quality  engineers  should 
mentor  project  team  members  and  listen  to  their  concerns  to  ensure  that  the  best  quality  processes 
are  utilized  and  best  quality  products  are  built.  Make  sure  that  there  is  a  two-way  street  for 
communications . 


SUMMARY 


Process  and  Product  Quality  Assurance  is  the  means  by  which  project  team  members,  as  well  as 
management,  get  insight  into  the  processes  used  and  work  products  produced  during  the  duration  of 
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a  product’s  development.  Small  projects  by  their  nature  have  proportionally  more  need  for  PPQA, 
yet  they  are  often  the  ones  that  can  least  afford  it.  Clearly,  it  is  important  for  the  resources  that  are 
dedicated  towards  small  projects  to  be  used  as  effectively  as  possible  because  the  margin  for  error  is 
lower  than  for  larger  projects. 

If  the  personnel  who  are  performing  PPQA  are  technically  qualified  to  do  the  type  of  work  they  are 
monitoring,  they  can  be  far  more  effective  in  performing  their  duties  than  those  who  are  not.  They 
are  more  likely  to  detect  problems  in  the  product,  and  identify  them  earlier,  than  someone  who  does 
not  understand  the  work  they  are  monitoring. 

Planning  PPQA  activities  can  take  proportionally  more  time  for  a  smaller  project  than  a  larger  one, 
so  it  is  desirable  to  streamline  the  planning  process  as  much  as  possible.  The  use  of  generic  plans 
and  schedule  templates  can  help  reduce  the  time  needed  to  plan  PPQA  activities. 

When  deciding  how  to  allocate  scarce  PPQA  resources  to  projects,  the  risks  must  be  evaluated  to 
decide  which  projects  need  more  of  those  resources.  The  technical  experience  level  of  the  project 
team  members,  their  history  of  process  compliance,  and  their  familiarity  with  the  specific  type  of 
product  being  developed  must  all  be  considered.  Each  project’s  potential  cost  to  the  organization  if 
it  fails  must  also  be  considered  when  determining  the  level  of  effort  that  should  be  allocated  for 
PPQA. 

Personnel  who  perform  PPQA  on  multiple  small  projects  are  in  a  unique  position  to  facilitate 
communication  if  most  developers  in  the  organization  normally  work  on  a  single  project  at  a  time. 
They  can  help  to  rapidly  spread  important  technical  information  between  the  project  teams.  They 
can  also  help  to  identify  experts  on  one  project  team  whose  expertise  might  be  extremely  valuable  in 
solving  a  problem  another  team  is  confronting. 

Even  in  small  doses,  the  presence  of  PPQA  on  a  project  reminds  the  team  that  there  is  a  process 
that  they  need  to  follow  as  they  develop  their  products,  and  there  are  certain  standards  that  those 
products  must  meet.  They  become  the  “little  voice”  in  the  minds  of  the  developers  and  act  as  the 
conscience  of  the  team  regarding  process  compliance. 
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Motivation  lor  tHis  Presentation 


Process  “failures”  have  been  identified  as  a  source  of 
program  problems 

-  By  DoD 

-  By  industry,  including  Lockheed  Martin 

Using  CMMI®  requires  (at  maturity  level  3)  that  processes 
tailored  from  the  organizational  standard  process  be 
deployed  on  programs 

However,  even  in  organizations  using  CMMI® 

-  The  “right”  process  isn’t  always  deployed 

-  The  process  isn’t  always  deployed  “right” 
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Agenda 


What  is  the  “right”  process  for  a  program? 


How  do  we  ensure  the  process  is  deployed 
“right”? 


What  is  tffle  “tigflt”  process? 

The  “right”  process 

•  Meets  requirements,  including  standards 

-  From  the  customer 

-  From  the  organization 

•  Is  tailored  from  the  organizational  standard  process 

•  Is  appropriately  suited  to  the  domain  and  program 

•  Contains  necessary  and  sufficient  process  elements 

•  Is  integrated  across  the  disciplines 


'fho  Lookhood  Martin  Iniograiod  dniorprioo  Prooooa  (LPJ-JPP) 
livlas  zhqulra/nanta  on  tho  Organizational  Standard  Proooddt 
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LM-IEP  In  Context 


LM-IEP  Architecture 


Enterprise 

Processes 


I 


Business  Execution  Processes 

Infrastructure 

Processes 


Common 

Management 

Processes 


Program  Execution  Processes 

Product  Life  Cycle  Processes 


A.1  Organizational 
Management 

A.2  Strategic  Planning 

A.3  Quality 

Management 

A.4  Ethics  &  Business 
Conduct 

A.5  Legal 

A.6  Communications 


B.1  Process  Management 

B.2  Work  Environment 
Management 

B.3  Technology  Mgmt 

B.4  Contracts 

B.5  Workforce  Mgmt 
B.6  Finance 

B.7  Supplier  Agreements 
and  Procurement 


C.1  Project  Planning 

C.2  Decision  Analysis 

C.3  Configuration  and 
Data  Management 

C.4  Performance 
Assessment  and 
Control 

C.5  Risk  and 
Opportunity 
Management 


D.1  Program  Mgmt 

D.2  Business 
Capture 

D.3  Development*" 

D.4  Production 

D.5  Deployment 

D.6  Operation  and 
Sustainment 

D.7  Disposal 


r 

D.3.1  Stakeholder 
Needs  Analysis 

D.3.2  Requirements 
Development 

D.3.3  Architectural 
Design 

D.3.4  Detailed  Design 

D.3.5  Implementation 
D.3.6  Integration 


B.8  Security 
B.9  Property  Mgmt 


D.3.7  Verification 


D.3.8  Validation 

V 

*  Recursive  processes  to  be 
applied  at  any  level  of  the 
system  hierarchy 
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Goal:  Consistent  OSP  Architectures 


LM-IEP  Architectural  Conformance 
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■ 
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Policy 
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etc. 
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Corporate 
Intellectual 
Capital  Collection 


Online  Access  to  Assets  —  LM-PAL 
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Key  Architectural  Tenets 

•  Architecture  covers  the  entire  enterprise 

-  To  be  used  as  a  taxonomy  for  Corporate  command  media 

^8  Detailed  taxonomy  below  IEP  level  to  be  determined  by 
responsible  functional  organizations 

•  Is  complete  in  scope,  not  in  requirements 

m  Requirements  based  on  source  standards,  thus  heavy  emphasis 
on  PM,  Quality,  and  Engineering 

-  Requirements  in  other  areas  need  to  be  augmented  by  existing 
corporate  policies  and  procedures,  and  other  industry  standards 

•  Represents  a  single  architectural  “view” 

-  Presents  process  elements  from  a  topical  viewpoint 

-  Other  views  required  for  management  and  practitioners; 
e.g.,  temporal,  role-based,  information  flow 
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Implementation 
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How  do  we  ensure  the  process  is 
deployed  “rignt”?  - 1 


The  typical  approach  involves.... 

•  Organizational  policy  (“thou  shalt...  ”) 

•  Process  &  Product  Quality  Assurance 

•  Mechanisms  for  ensuring  process  fidelity,  including 

-  Process-enforcement  mechanisms  such  as  process 
enactment  tools 

-  Process  tailoring  approval 

-  Quality  assurance  audits 

-  Reviews,  checklists,  etc. 
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How  do  we  ensure  the  process  is 
deployed  “rignt”?  -  2 


•  Lockheed  Martin  experience  is  that  ensuring  the 
process  is  deployed  “right”  requires 

-  Process  checkpoints  synchronized  with  a  program’s 
business  rhythm 

-  Including  process  improvement  investment  during 
strategic,  long-range  planning 

-  Prescribing  organizational  participation  in  corporate- 
level  infrastructure 
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Corporate  Policy  Statement  (CPS)  on 
Program  Management  (PM) 

•  A:  Program  Manager  Development 

•  B:  Risk  Management 

•  C:  Past  Performance 

•  D:  Proposal  and  Program  Reviews  Updated 

•  E:  Data  Management 

•  F:  Configuration  Management 

•  G:  Managing  Major  Subcontracts 

•  H:  Integrated  Planning  &  Scheduling  New 

•  I:  Program  Performance  Reporting  New 

Corporate  Direction  to  Formalize  the  PM  Infrastructure 
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Synchronizing  Process  Checkpoints^^ 
with  a  Program’s  Business  Rflythm 

Pursue  RFP  Submit  Award 

1  1 

~  Pre-P 

Strategy 
Development 

roposal 

Opportunity 

Positioning 

▼  ▼ 

Proposal  Post-Proposal  Program 

Development  Substantiation  Performance 

Customer  Events 

RFI 

DRFP  Release  FRFP  Proposal  Contract 

Release  Submit  Award 

LM  Color 

Reviews 

Black 

Hat(s) 

A  A. 

Pre-Proposal  Blue  Team 

Kaizen  (PPK) 

A 

Red  Team 

New  Business  ^ 

Opportunity 

Management  (CPS-009) 

PT\A/  9  Pr-Li-n 

k  ▲ 

r  1  vv  &  oom 

oetition  Analysis 

▲4 - 

-  Proposal  - ►  ▲ 

Program  Management 

CPS-070 

Sub-Contract 
Management  Plan 

Program 

RM  Plan  Startup _ 

Q^p  _ w 

Months 

Common  Checklist  for  Program  Reviews  , 

II  15 — ‘ 

Independent  Cost  Est 

C  PS-440 

ICE 

Approval  of  Proposals 

CPS-017 

EPP 

Allows: 


•  On-line  Completion  and  Storage  of  Checklist 

•  Centralized  Repository  for  Review  Artifacts 

•  Automatic  Action  Item  Generation 

•  Summary  Metrics  of  Checklist  Findings 
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Assuring  Organizational  and  Program- 
Process  Compliance  &  Implementation 


Customer 

Appraisals 


Contractor 

Appraisals 


Command 

Media 


Program 


Recommendation: 


Institute  the  requirement  for  process  maturity  in  command  media 
with  check  and  balance  for  implementation  and  management. 
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Future  Improvements 


Fully  electronic  processes  (using  models/tools)  to 

-  Improve  human  understanding  and  communication 

-  Improve  process  fidelity,  management  and 
improvement 

-  Implement  multiple  views  (e.g.,  behavioral, 
functional,  organizational,  informational) 

•  Support  process  enactment 

Improved  program  startup 

-  Ensure  smooth  transition  from  proposal  phase 

-  Enable  quick  and  robust  program  initiation 


Process  Improvement  Strategy 

Example 


1/06 

IEP  V3  Gap 
Analysis 


6/06  IEP  V3  Gaps 
closed 

AS9100  certification 

4/06  Plan  for  Electronic 
Process  Management  System 


8/06 

CM 

Gap 


MI  V1.2 
Analysis 


LM-IEP  architecture  transition  strategy 

4  LM-IEP  V3  Requirements  gap  closure 
AS9100  certification 


-+  6/06 


10//06 
CMMI  VI  .2 
Gaps  closed 

9/06  IMP/IMSs  process 
deployment  complete 


CMMI  VI. 2  gap  analysis  and  closure 
◄ - — - ►  10/06 


12/07  SCAMPI, 
Maturity  Level  4 


6/07  IDE 
implementation 


EP 


vlanagement 
mented 


nont  ei/ctopfi 

fc/HQ 


mem 


-> 


JMP/IMS  process  update  implementation  and  training 


->  9/0 


haital 


mpl 


■> 
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EPI  Program  Infrastructure:  2005 


Supportability 


PWB  / 
CCA 


Joint  Working  Groups  (all  SCs  supporting) 

•  Integrated  Measurement  (EPM  led) 

•  Integrated  Sys.  Dev.  Methodology  (SESC  led) 


j 


•  Integration  an  Test  (SESC  led) 


Subcouncils 

•  Consensus  on  best 
practices,  process, 
methods,  tools 

Process  Groups 

•  High  Priority  initiatives 


•  Tool  Integration  (EPI  led) 
Maintain  Processes,  Select  Standard  Tools 

1  t  1 


Subcouncil  /  Process  Group  Representatives 
•  Implement  EPI  Processes 
•  Apply  Standard  Tools 


t 


Provides  Engineering 

4 -  Support  for 

ProcessATools 


SubCouncil/Process  Group  Participation  Benefits  Members,  Companies  and  Lockheed  Martin 
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Summary 


Selecting  the  “right”  process  for  a  program  is  non¬ 
trivial  and  requires 

-  Having  the  “right”  OSP 

-  Using  the  “right”  assets  to  support  the  process 

Supporting  infrastructure  facilitates  deploying  the 
process  “right” 

-  Process  checkpoints  linked  to  program  milestones 

-  Strategic  investment  to  leverage  across  businesses 

-  Infrastructure  support  (e.g.,  participation  in 
corporate-level  councils) 


LM-IEP  Standard 


AS9100  to  LM-IEP 


Defines  “what”  level  of  processes  versus  “how” 

•  LM-IEP  Standard  specifies  what  is  to  be 
performed 

•  Company  processes,  methods  and  tools 
specify  how 


5.  List  of 
Acronyms 


IEEE  1220  to 
LM-IEP 


EPI  280-07  to 
LM-IEP 


ISO/IEC  15288  to 
LM-IEP 


ISO/IEC  12207  to 
LM-IEP 


4.  Glossary 


3.  LM-IEP 
Conformance 


2.  Referenced 


ISO  9001 :2000  to 
LM-IEP 


C  M  M  l®-S  E/S  W/l  P  P  D/ 
SS  to  LM-IEP 


Process 


EIA/ANSI  632  to 
LM-IEP 


PART  I 
OVERVIEW 


PART  II 

PROCESS 

SPECIFICATIONS 


PART  III 

TRACEABILITY  TO 

SOURCE 

STANDARDS 


Documents 

i  upuomoaiiui  io 

LM-IEP  To  Source 
Standards 

|  1 .  Introduction 

Preface 

LM-IEP  Requirements 
Conformance  Matrix 


PART  IV 
LM-IEP 

CONFORMANCE 

MATRICES 


Standard  (280-01)  available  on  the  EPI  web-site  at 
http://www.epic.lmco.com/docs/280-01all/index.htm 
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Capability  Maturity  Model  Integration 

Technology  Conferee 

User  Group 


JJaulenanjUGeneraJ  Joseph  L.  Yakovac,  Jr. 
_  Military  Deputy  to  the 


Assistant  Secretary  of  the  Army 


Denver,  Colorado 
15  November  2005 


m 


Future  force  Capabilities 


The  Future  Force 


Future  Force  Characteristics  - 
Responsive,  Deployable,  Agile,  Versatile, 
Lethal,  Survivable,  Sustainable... 

A  New  Way  of  Joint  Warfare 

Dominant  situation  awareness 
Networked  weapons  systems 
Joint  Interdependence  to  Small  Unit  Level 

More  Strategically  Responsive  Land  Force 

**  Lighter,  more  air  and  sea  transportable 
Reduced  sustainment  footprint/  reachback/3 
days  combat  without  re-supply 

Technology  Enabled  -  Spiral 
development/insertions 

Capabilities  Based  Force  for  combatant 
commanders  now...  future! 


•  _  _  s  ■  mm  11  w  i  y  — r— .  — y: .1^— =  m _ 

If  Capability  Maturity  Model  Integration 
(CMMI)  reduces  the  costs  of  systems 


-engineering  components, -then  what  is  th& 
next  step  to  ensurefhat  svsterfi  6f*svstemsi 


engineering improves  cost,  schedule,  and 

overall  quality? 


are  systems  engineers? 


How  do  you  assess  thelCmivillvff 
level  of  a  system-  of  systems  that 
-brings-together-a  g ro u p  of  users  k 
— and-developers  who  normally 
focus  on  stand-alone  programs? 


